From df1893c0ece185151f66522a75f85368ba9bfd26 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 19:58:35 -0400 Subject: [PATCH 01/94] Start #578 increment 2 progress ledger --- PROGRESS.md | 66 ++++++++++++++++++++++------------------------------- 1 file changed, 27 insertions(+), 39 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index f3158fdd..f700a804 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,48 +1,36 @@ -# Progress +# Progress: populace#578 increment 2 ## State -Populace #516 whole-row donor outlier screen is complete on -`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515 -interim carve merged as #525). The `puf_tax_detail` donor now drops tax units -whose grouped raw mortgage interest reaches $10M before the #515 carve -(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T -of phantom mortgage-interest mass), with the checkpoint schema bumped to v3 -so post-carve pre-screen checkpoints rebuild. +- Branch: `multispine-pool-build-578` +- Worktree: `/Users/maxghenis/PolicyEngine/_worktrees/populace-578-inc2` +- Scope: code and small synthetic-fixture tests only; no dataset downloads or + full-data builds. +- Remote `main` was verified through GitHub at merge commit + `6c14a0a8590402d1805a24e55ca5f017f39dc281`. +- The sandbox blocked `git fetch` at DNS resolution. The local worktree is + temporarily rooted at merged PR #581's exact head + `956dc0a3dd5ceaffae0e2007d98dc951e773d389`, whose tree is the merge result. + Rebase onto fetched `origin/main` remains a handoff prerequisite if network + access is not restored in this lane. ## Done -- Confirmed a clean starting worktree at `aef1c56`. -- Read the repository guidance and established the #515 donor carve as the - screen's required downstream boundary. -- Started source-level audits of every donor-frame consumer, checkpoint - validation, row-count pins, and existing donor-fact summaries. -- Attempted the requested GitNexus impact workflow; the managed filesystem - denied its global registry write. Its local index also exposed a broad - `build/` ignore mismatch, so the completed impact audit uses direct source - call sites and tests. -- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the - structural rationale and pinned-artifact receipts. -- Added a whole-row screen on grouped raw person `home_mortgage_interest` - after tax-unit assembly, before the #515 carve, with retained-index reset. -- Confirmed no downstream consumer pairs donor rows to the original HDF arrays - or carries a stale donor-length vector; values and weights always originate - from the same screened frame. -- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale - checkpoint regression track the live constant while retaining literal-v1 - corruptions. -- Added regression coverage for the exact grouped boundary, whole-row removal, - retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant. -- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets - 139; populace-data 138 with 1 skip. The directly affected tail-bound suite - adds 12 passes. Ruff format/check and `git diff --check` are clean. -- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line - audit, expected 208,611-row real-artifact effect, verification results, count - sweep, and deliberately untouched surfaces. +- Read populace#578, its governing “Scope hardening” section, and the UK parity + audit comment. +- Read merged populace#581's contract, review note, changed-file inventory, and + merge metadata. +- Verified GitHub's current `main` tip is `6c14a0a`. +- Created the requested branch and worktree without modifying the existing + checkout. ## Next -- PR #527 review cycle, then merge. After both #525 and #527: rebuild the - base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a - run that holds per `us_critical_targets.py`. -- Root record-level ETL carve stays open on populace#515. +1. Map the pre-clone ASEC product, ACS unit-frame builder, assembly/clone/gate + contracts, and existing multispine tool consumers. +2. Design the canonical assemble → clone → impute/derive/seed → agreement + pipeline with sha-pinned explicit inputs and failure receipts. +3. Implement the pool builder and retain the old tool only as a thin deprecated + shim if consumers require it. +4. Add synthetic full-path tests, changelog fragment, focused verification, and + the external review worklog. From 089fe96c28de80214d5e04657b81cca6f8172b3a Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 20:11:27 -0400 Subject: [PATCH 02/94] Record increment 2 wiring design --- PROGRESS.md | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index f700a804..ab6f0cc7 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -23,14 +23,26 @@ - Verified GitHub's current `main` tip is `6c14a0a`. - Created the requested branch and worktree without modifying the existing checkout. +- Audited the legacy ACS multispine tool's consumers. The local-release builder + imports its H5 helpers and the published build recipe names its CLI, so the + legacy path will remain only as a deprecated compatibility shim. +- Traced the production stage contracts from the ASEC pre-clone checkpoint and + byte-pinned ACS PUMS acquisition through assembly, PUF cloning and transfer, + derivation, seeding, SSI simulation, and the fixed agreement gate. +- Identified two correctness fixes required by the new ordering: + `transfer_acs_inputs` must fill missing cells without overwriting measured + values, and every post-assembly `Frame` reconstruction must preserve the + #581 metadata receipt. +- Ran the focused #581 assembly, agreement, clone-routing, and AST-guard tests + against the starting tree: 44 passed. ## Next -1. Map the pre-clone ASEC product, ACS unit-frame builder, assembly/clone/gate - contracts, and existing multispine tool consumers. -2. Design the canonical assemble → clone → impute/derive/seed → agreement - pipeline with sha-pinned explicit inputs and failure receipts. -3. Implement the pool builder and retain the old tool only as a thin deprecated - shim if consumers require it. +1. Implement missing-cell/raw-preserving transfer and receipt-safe + post-assembly stages. +2. Add the canonical pool orchestrator and sha-pinned CLI with deterministic + manifest/diagnostic paths and no tolerance knobs. +3. Convert the legacy CLI to a deprecated shim while preserving the helper + imports its known consumer needs. 4. Add synthetic full-path tests, changelog fragment, focused verification, and the external review worklog. From e0d1f6ee654f0f9aa58acdbf68a628224c6291ed Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 20:23:50 -0400 Subject: [PATCH 03/94] Add ordered multispine pool runtime seam --- PROGRESS.md | 16 +- .../build/us_runtime/multispine_pool.py | 226 ++++++++++++++ .../build/us_runtime/support_provenance.py | 83 ++++++ .../tests/test_us_multispine_pool.py | 275 ++++++++++++++++++ .../tests/test_us_spine_blindness.py | 1 + 5 files changed, 598 insertions(+), 3 deletions(-) create mode 100644 packages/populace-build/src/populace/build/us_runtime/multispine_pool.py create mode 100644 packages/populace-build/tests/test_us_multispine_pool.py diff --git a/PROGRESS.md b/PROGRESS.md index ab6f0cc7..393d7b0d 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -35,12 +35,22 @@ #581 metadata receipt. - Ran the focused #581 assembly, agreement, clone-routing, and AST-guard tests against the starting tree: 44 passed. +- Added the canonical order-bearing runtime seam: + assemble → clone → impute → derive → seed → simulate → terminal agreement. + The simulated formula-output view is separate from the returned input-only + pool, and every operator boundary revalidates the immutable #581 receipt. +- Added provenance-owner reporting helpers for JSON-ready assembly receipts and + per-entity source-channel/clone-index counts; population operators still + receive no source-routing interface. +- Added small two-source tests for the ordered path, batched red agreement, + receipt loss, clone-safe ID refusal, manifest counts, and default take-up + inventory coverage. The new seam and the full spine-blindness guard pass. ## Next -1. Implement missing-cell/raw-preserving transfer and receipt-safe - post-assembly stages. -2. Add the canonical pool orchestrator and sha-pinned CLI with deterministic +1. Finish missing-cell/raw-preserving transfer and receipt-safe production + operators. +2. Add the sha-pinned CLI with deterministic manifest/diagnostic paths and no tolerance knobs. 3. Convert the legacy CLI to a deprecated shim while preserving the helper imports its known consumer needs. diff --git a/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py b/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py new file mode 100644 index 00000000..e0618799 --- /dev/null +++ b/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py @@ -0,0 +1,226 @@ +"""Canonical pre-calibration build path for the assembled US spine pool. + +The path is intentionally small and order-bearing: + +``assemble -> clone -> impute -> derive -> seed -> simulate -> agreement``. + +Assembly owns source provenance. Every later operator receives the whole pool +and must be source-spine blind; PUF-detail routing is clone-index based. The +simulation copy exists only to evaluate formula-owned agreement outputs and is +not the input-only pool returned for H5 publication. +""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping +from dataclasses import dataclass, field + +from populace.build.gates import GateResult +from populace.build.us_runtime.acs_transfer import ( + TargetFamilies, + declared_acs_transfer_target_families, +) +from populace.build.us_runtime.puf_support import clone_us_frame_for_puf_support +from populace.build.us_runtime.spine_agreement import spine_agreement_gate +from populace.build.us_runtime.spine_assembly import assemble_spines +from populace.build.us_runtime.support_provenance import ( + spine_assembly_receipt, + spine_provenance_counts, + validate_assembly_provenance, +) +from populace.build.us_runtime.take_up_contract import load_take_up_contract +from populace.frame import Frame + +__all__ = [ + "POOL_HOUSEHOLD_MASS_SHARES", + "POOL_OPERATOR_ORDER", + "MultispinePoolResult", + "PoolStageOutput", + "pool_transfer_target_families", + "run_multispine_pool_path", +] + +POOL_HOUSEHOLD_MASS_SHARES: Mapping[str, float] = { + "asec": 0.5, + "acs": 0.5, +} +"""Fixed peer-spine mass shares; calibration remains downstream.""" + +POOL_OPERATOR_ORDER = ( + "assemble", + "clone", + "impute", + "derive", + "seed", + "simulate", + "agreement", +) +"""The executable pool-build order, including the terminal QA evaluation.""" + + +@dataclass(frozen=True) +class PoolStageOutput: + """One source-blind operator result and its manifest-ready receipt.""" + + frame: Frame + receipt: Mapping[str, object] = field(default_factory=dict) + + def __post_init__(self) -> None: + if not isinstance(self.frame, Frame): + raise TypeError( + "PoolStageOutput.frame must be a Frame, got " + f"{type(self.frame).__name__}." + ) + if not isinstance(self.receipt, Mapping): + raise TypeError("PoolStageOutput.receipt must be a mapping.") + + +@dataclass(frozen=True) +class MultispinePoolResult: + """Input-only pool plus receipts from its terminal agreement evaluation.""" + + frame: Frame + assembly_receipt: Mapping[str, object] + provenance_counts: Mapping[str, Mapping[str, object]] + stage_receipts: Mapping[str, Mapping[str, object]] + agreement_gate: GateResult + + @property + def simulation_ready(self) -> bool: + """Whether the unchanged terminal agreement gate passed.""" + + return self.agreement_gate.passed + + +type PoolOperator = Callable[[Frame], PoolStageOutput] +type AgreementGate = Callable[[Frame], GateResult] + + +def pool_transfer_target_families() -> TargetFamilies: + """Return the fixed pool transfer plan checked by the #581 gate. + + The existing ACS QRF declaration remains the base. Every take-up input in + the checked-in contract is added when that declaration does not already + own it, because the ACS peer has no measured take-up flags and the terminal + default agreement registry requires the complete take-up surface. + """ + + plan = { + entity: {family: tuple(columns) for family, columns in families.items()} + for entity, families in declared_acs_transfer_target_families().items() + } + owned = { + (entity, column) + for entity, families in plan.items() + for columns in families.values() + for column in columns + } + additions: dict[str, list[str]] = {} + for program in load_take_up_contract().programs: + key = (program.entity, program.variable) + if key in owned: + continue + additions.setdefault(program.entity, []).append(program.variable) + owned.add(key) + for entity, columns in additions.items(): + families = plan.setdefault(entity, {}) + existing = families.get("take_up", ()) + families["take_up"] = (*existing, *sorted(columns)) + return plan + + +def run_multispine_pool_path( + asec: Frame, + acs: Frame, + *, + impute: PoolOperator, + derive: PoolOperator, + seed: PoolOperator, + simulate: PoolOperator, + agreement_gate: AgreementGate | None = None, +) -> MultispinePoolResult: + """Run the fixed assembly-to-agreement path over two peer source frames. + + ``impute``, ``derive``, and ``seed`` each receive the entire cloned pool. + They have no source label argument and are checked at their output boundary + against the immutable assembly receipt. ``simulate`` returns a temporary + evaluation frame: formula-owned outputs on that copy are visible to the + terminal gate but never enter :attr:`MultispinePoolResult.frame`. + + ``agreement_gate`` is an injection seam for small synthetic tests only. + Production callers omit it, which invokes + :func:`~populace.build.us_runtime.spine_agreement.spine_agreement_gate` + with its fixed registry and tolerances. + """ + + operators = { + "impute": impute, + "derive": derive, + "seed": seed, + "simulate": simulate, + } + invalid = [name for name, operator in operators.items() if not callable(operator)] + if invalid: + raise TypeError(f"Pool operator(s) are not callable: {invalid}.") + + assembled = assemble_spines( + {"asec": asec, "acs": acs}, + household_mass_shares=POOL_HOUSEHOLD_MASS_SHARES, + mass_anchor_channel="asec", + ) + assembly_receipt = spine_assembly_receipt( + assembled, + boundary="multispine pool assembly", + ) + + current = clone_us_frame_for_puf_support(assembled) + validate_assembly_provenance( + current, + boundary="multispine pool clone output", + ) + + receipts: dict[str, Mapping[str, object]] = {} + for stage_name in ("impute", "derive", "seed"): + outcome = operators[stage_name](current) + if not isinstance(outcome, PoolStageOutput): + raise TypeError( + f"Pool {stage_name} operator must return PoolStageOutput, got " + f"{type(outcome).__name__}." + ) + current = outcome.frame + validate_assembly_provenance( + current, + boundary=f"multispine pool {stage_name} output", + ) + receipts[stage_name] = dict(outcome.receipt) + + counts = spine_provenance_counts( + current, + boundary="multispine pool pre-agreement output", + ) + simulated = operators["simulate"](current) + if not isinstance(simulated, PoolStageOutput): + raise TypeError( + "Pool simulate operator must return PoolStageOutput, got " + f"{type(simulated).__name__}." + ) + validate_assembly_provenance( + simulated.frame, + boundary="multispine pool simulation output", + ) + receipts["simulate"] = dict(simulated.receipt) + + gate_operator = spine_agreement_gate if agreement_gate is None else agreement_gate + agreement = gate_operator(simulated.frame) + if not isinstance(agreement, GateResult): + raise TypeError( + "Pool agreement operator must return GateResult, got " + f"{type(agreement).__name__}." + ) + return MultispinePoolResult( + frame=current, + assembly_receipt=assembly_receipt, + provenance_counts=counts, + stage_receipts=receipts, + agreement_gate=agreement, + ) diff --git a/packages/populace-build/src/populace/build/us_runtime/support_provenance.py b/packages/populace-build/src/populace/build/us_runtime/support_provenance.py index b67381b8..d5d68067 100644 --- a/packages/populace-build/src/populace/build/us_runtime/support_provenance.py +++ b/packages/populace-build/src/populace/build/us_runtime/support_provenance.py @@ -16,6 +16,8 @@ "has_support_role_metadata", "puf_tax_detail_clone_mask", "spine_assembly_manifest", + "spine_assembly_receipt", + "spine_provenance_counts", "spine_source_id_column", "support_channel_column", "support_clone_index_column", @@ -242,6 +244,70 @@ def validate_assembly_provenance( return manifest +def spine_assembly_receipt( + frame: _ProvenanceFrame, + *, + boundary: str, +) -> dict[str, object]: + """Return a mutable, JSON-ready copy of a validated assembly receipt. + + :class:`~populace.frame.Frame` freezes metadata recursively. Build + manifests need ordinary dictionaries and lists, but must only publish a + receipt after validating it against the live provenance columns. + """ + + manifest = validate_assembly_provenance(frame, boundary=boundary) + if manifest is None: # pragma: no cover - require_manifest defaults true + raise AssertionError("Validated assembly receipt unexpectedly absent.") + return _json_ready_mapping(manifest) + + +def spine_provenance_counts( + frame: _ProvenanceFrame, + *, + boundary: str, +) -> dict[str, dict[str, object]]: + """Count every source channel and clone index without exposing routing. + + This reporting helper deliberately lives with the provenance owner. + Population operators remain unable to branch on source identity; manifests + can still publish a complete per-entity receipt after the shared validator + proves the live columns agree with assembly. + """ + + manifest = validate_assembly_provenance(frame, boundary=boundary) + if manifest is None: # pragma: no cover - require_manifest defaults true + raise AssertionError("Validated assembly receipt unexpectedly absent.") + channels = tuple(str(channel) for channel in manifest["channels"]) + counts: dict[str, dict[str, object]] = {} + for entity in frame.entities: + table = frame.table(entity) + source = table[support_channel_column(entity)].astype(str) + clone_index = pd.to_numeric( + table[support_clone_index_column(entity)], + errors="raise", + ).astype("int64") + observed_clone_indices = sorted(int(value) for value in clone_index.unique()) + counts[entity] = { + "rows": int(len(table)), + "by_source_channel": { + channel: int(source.eq(channel).sum()) for channel in channels + }, + "by_clone_index": { + str(index): int(clone_index.eq(index).sum()) + for index in observed_clone_indices + }, + "by_source_channel_and_clone_index": { + channel: { + str(index): int((source.eq(channel) & clone_index.eq(index)).sum()) + for index in observed_clone_indices + } + for channel in channels + }, + } + return counts + + def has_support_role_metadata( table: pd.DataFrame, *, @@ -426,3 +492,20 @@ def puf_tax_detail_clone_mask( def _require_entity_name(entity: str) -> None: if not isinstance(entity, str) or not entity: raise ValueError("entity must be a non-empty string.") + + +def _json_ready_mapping(value: Mapping[str, Any]) -> dict[str, object]: + """Deep-copy frozen receipt values into JSON-compatible containers.""" + + def thaw(item: Any) -> Any: + if isinstance(item, Mapping): + return {str(key): thaw(nested) for key, nested in item.items()} + if isinstance(item, tuple): + return [thaw(nested) for nested in item] + if isinstance(item, list): + return [thaw(nested) for nested in item] + if isinstance(item, np.generic): + return item.item() + return item + + return {str(key): thaw(item) for key, item in value.items()} diff --git a/packages/populace-build/tests/test_us_multispine_pool.py b/packages/populace-build/tests/test_us_multispine_pool.py new file mode 100644 index 00000000..5b68ccee --- /dev/null +++ b/packages/populace-build/tests/test_us_multispine_pool.py @@ -0,0 +1,275 @@ +from __future__ import annotations + +from collections.abc import Callable + +import numpy as np +import pandas as pd +import pytest + +from populace.build.gates import GateResult +from populace.build.us_runtime.multispine_pool import ( + POOL_OPERATOR_ORDER, + PoolStageOutput, + pool_transfer_target_families, + run_multispine_pool_path, +) +from populace.build.us_runtime.puf_support import ( + PUF_SUPPORT_MAX_CLONE_SAFE_SOURCE_ID, +) +from populace.build.us_runtime.spine_agreement import ( + SpineAgreementSpec, + spine_agreement_gate, +) +from populace.frame import US_SCHEMA, Frame, WeightKind, Weights + + +def _source_frame(*, offset: float = 0.0) -> Frame: + ids = np.asarray([1, 2], dtype=np.int64) + person = pd.DataFrame( + { + "person_id": ids, + "person_household_id": ids, + "person_tax_unit_id": ids, + "person_spm_unit_id": ids, + "person_family_id": ids, + "person_marital_unit_id": ids, + "age": np.asarray([30.0, 50.0]), + "measured": np.asarray([1.0, 2.0]) + offset, + } + ) + tables = { + "person": person, + **{ + entity: pd.DataFrame({f"{entity}_id": ids}) + for entity in US_SCHEMA.group_entities + }, + } + return Frame( + tables, + US_SCHEMA, + { + "household": Weights( + np.asarray([2.0, 2.0]), + WeightKind.DESIGN, + ) + }, + pd.Series(["fixture", "fixture"], dtype=object), + ) + + +def _replace_person( + frame: Frame, person: pd.DataFrame, *, metadata: bool = True +) -> Frame: + tables = {entity: frame.table(entity).copy() for entity in frame.entities} + tables["person"] = person + return Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata if metadata else None, + ) + + +def _operator( + name: str, + order: list[str], + transform: Callable[[pd.DataFrame], None], +) -> Callable[[Frame], PoolStageOutput]: + def apply(frame: Frame) -> PoolStageOutput: + order.append(name) + person = frame.table("person").copy() + transform(person) + return PoolStageOutput( + _replace_person(frame, person), + {"operator": name}, + ) + + return apply + + +def _fixture_registry() -> tuple[SpineAgreementSpec, ...]: + return ( + SpineAgreementSpec("person", "imputed", ("transferred",)), + SpineAgreementSpec("person", "derived", ("derived",)), + SpineAgreementSpec("person", "take_up", ("seeded",)), + SpineAgreementSpec("person", "simulated_output", ("ssi",)), + ) + + +def test_full_operator_path_is_ordered_and_keeps_simulation_out_of_pool() -> None: + order: list[str] = [] + impute = _operator( + "impute", + order, + lambda person: person.__setitem__("transferred", person["age"]), + ) + derive = _operator( + "derive", + order, + lambda person: person.__setitem__("derived", person["transferred"] * 2), + ) + seed = _operator( + "seed", + order, + lambda person: person.__setitem__("seeded", person["age"] >= 40), + ) + simulate = _operator( + "simulate", + order, + lambda person: person.__setitem__("ssi", person["derived"]), + ) + + result = run_multispine_pool_path( + _source_frame(), + _source_frame(), + impute=impute, + derive=derive, + seed=seed, + simulate=simulate, + agreement_gate=lambda frame: spine_agreement_gate( + frame, + registry=_fixture_registry(), + ), + ) + + assert tuple(["assemble", "clone", *order, "agreement"]) == POOL_OPERATOR_ORDER + assert order == ["impute", "derive", "seed", "simulate"] + assert result.agreement_gate.passed + assert result.simulation_ready + assert "ssi" not in result.frame.table("person") + assert result.assembly_receipt["channels"] == ["asec", "acs"] + assert result.assembly_receipt["native_row_counts"]["person"] == { + "asec": 2, + "acs": 2, + } + assert result.provenance_counts["person"] == { + "rows": 8, + "by_source_channel": {"asec": 4, "acs": 4}, + "by_clone_index": {"0": 4, "1": 4}, + "by_source_channel_and_clone_index": { + "asec": {"0": 2, "1": 2}, + "acs": {"0": 2, "1": 2}, + }, + } + assert result.stage_receipts == { + "impute": {"operator": "impute"}, + "derive": {"operator": "derive"}, + "seed": {"operator": "seed"}, + "simulate": {"operator": "simulate"}, + } + + +def test_agreement_failures_remain_batched_in_terminal_result() -> None: + order: list[str] = [] + + def imputed(person: pd.DataFrame) -> None: + person["transferred"] = person["measured"] + + def derived(person: pd.DataFrame) -> None: + person["derived"] = person["transferred"] + + def seeded(person: pd.DataFrame) -> None: + person["seeded"] = person["measured"] > 0 + + def simulated(person: pd.DataFrame) -> None: + person["ssi"] = person["measured"] + + result = run_multispine_pool_path( + _source_frame(), + _source_frame(offset=99.0), + impute=_operator("impute", order, imputed), + derive=_operator("derive", order, derived), + seed=_operator("seed", order, seeded), + simulate=_operator("simulate", order, simulated), + agreement_gate=lambda frame: spine_agreement_gate( + frame, + registry=_fixture_registry(), + ), + ) + + assert not result.agreement_gate.passed + assert not result.simulation_ready + assert len(result.agreement_gate.failures) >= 3 + assert result.agreement_gate.details["tolerances"] == { + "incidence_ratio_bounds": [0.8, 1.25], + "max_quantile_envelope_distance": 0.25, + } + assert order == ["impute", "derive", "seed", "simulate"] + + +def test_operator_metadata_drop_surfaces_assembly_receipt_error() -> None: + def no_op(frame: Frame) -> PoolStageOutput: + return PoolStageOutput(frame) + + def drop_receipt(frame: Frame) -> PoolStageOutput: + person = frame.table("person").copy() + return PoolStageOutput(_replace_person(frame, person, metadata=False)) + + with pytest.raises( + ValueError, + match="multispine pool derive output:.*no assembly manifest", + ): + run_multispine_pool_path( + _source_frame(), + _source_frame(), + impute=no_op, + derive=drop_receipt, + seed=no_op, + simulate=no_op, + agreement_gate=lambda _frame: GateResult("fixture", True), + ) + + +def test_clone_safe_id_violation_surfaces_assembly_error() -> None: + oversized = PUF_SUPPORT_MAX_CLONE_SAFE_SOURCE_ID + 1 + asec = _source_frame() + tables = {entity: asec.table(entity).copy() for entity in asec.entities} + tables["person"].loc[0, "person_id"] = oversized + asec = Frame( + tables, + asec.schema, + {"household": asec.weights_for("household")}, + asec.strata, + ) + + def unreachable(_frame: Frame) -> PoolStageOutput: + raise AssertionError("Assembly violations must precede every operator.") + + with pytest.raises(ValueError, match="Spine 'asec'.*clone-safe bound"): + run_multispine_pool_path( + asec, + _source_frame(), + impute=unreachable, + derive=unreachable, + seed=unreachable, + simulate=unreachable, + ) + + +def test_pool_transfer_plan_covers_every_take_up_without_duplicates() -> None: + families = pool_transfer_target_families() + ownership: dict[str, tuple[str, str]] = {} + for entity, by_family in families.items(): + for family, columns in by_family.items(): + for column in columns: + assert column not in ownership + ownership[column] = (entity, family) + + expected = { + "takes_up_snap_if_eligible", + "takes_up_tanf_if_eligible", + "takes_up_eitc", + "takes_up_medicaid_if_eligible", + "takes_up_chip_if_eligible", + "takes_up_basic_health_program_if_eligible", + "takes_up_medicare_if_eligible", + "takes_up_ssi_if_eligible", + "takes_up_dc_ptc", + "takes_up_head_start_if_eligible", + "takes_up_early_head_start_if_eligible", + "takes_up_housing_assistance_if_eligible", + "takes_up_aca_if_eligible", + } + assert expected <= set(ownership) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index f9f129fe..d94245ab 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -68,6 +68,7 @@ "form_4952.py", "housing_inputs.py", "medicare_take_up.py", + "multispine_pool.py", "other_health_insurance.py", "prior_year_income.py", "qbi_inputs.py", From bc2c197505468257af34331ba8f06d492b1839e4 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 20:30:49 -0400 Subject: [PATCH 04/94] Add bounded PUF donor artifact loader --- .../src/populace/build/us_runtime/__init__.py | 2 + .../populace/build/us_runtime/puf_donor_io.py | 75 +++++++++ .../tests/test_us_puf_donor_io.py | 147 ++++++++++++++++++ .../tests/test_us_spine_blindness.py | 1 + 4 files changed, 225 insertions(+) create mode 100644 packages/populace-build/src/populace/build/us_runtime/puf_donor_io.py create mode 100644 packages/populace-build/tests/test_us_puf_donor_io.py diff --git a/packages/populace-build/src/populace/build/us_runtime/__init__.py b/packages/populace-build/src/populace/build/us_runtime/__init__.py index aadcc28e..1fec8d6a 100644 --- a/packages/populace-build/src/populace/build/us_runtime/__init__.py +++ b/packages/populace-build/src/populace/build/us_runtime/__init__.py @@ -516,6 +516,7 @@ validate_puf_capital_gains_tail_manifest, write_puf_capital_gains_tail_manifest, ) +from populace.build.us_runtime.puf_donor_io import load_puf_tax_unit_donor from populace.build.us_runtime.puf_e01000_reconciliation import ( PUF_E01000_RECONCILIATION_SCHEMA_VERSION, build_puf_e01000_reconciliation_basis, @@ -1795,6 +1796,7 @@ "load_default_reform_specs", "load_congressional_district_vintage_crosswalk", "load_default_congressional_district_vintage_crosswalk", + "load_puf_tax_unit_donor", "normalize_district_code", "parse_baf_cd_layer", "parse_national_cd_bef_districts", diff --git a/packages/populace-build/src/populace/build/us_runtime/puf_donor_io.py b/packages/populace-build/src/populace/build/us_runtime/puf_donor_io.py new file mode 100644 index 00000000..2e2012d8 --- /dev/null +++ b/packages/populace-build/src/populace/build/us_runtime/puf_donor_io.py @@ -0,0 +1,75 @@ +"""Bounded file loading for the processed-PUF tax-unit donor. + +This module owns only the input boundary: it reads the root arrays from one +explicit processed PUF HDF5 file, aligns them to E00100 from one explicit +source-year PUF CSV, and delegates donor construction to the shared runtime. +It does not acquire data, choose a donor source, or relax the source-year +alignment checks. +""" + +from __future__ import annotations + +from pathlib import Path + +import numpy as np +import pandas as pd + +from populace.build.us_runtime.puf_source_agi import ( + source_year_puf_adjusted_gross_income, +) +from populace.build.us_runtime.puf_support import puf_tax_unit_donor_from_arrays + +__all__ = ["load_puf_tax_unit_donor"] + + +def load_puf_tax_unit_donor( + processed_puf_h5: str | Path, + source_year_puf_csv: str | Path | None, + *, + donor_build_summary: dict[str, object] | None = None, +) -> pd.DataFrame: + """Load a processed PUF donor with source-year E00100 alignment. + + Args: + processed_puf_h5: Explicit processed PUF HDF5 artifact. Every + root-level dataset is passed unchanged to + :func:`puf_tax_unit_donor_from_arrays`. + source_year_puf_csv: Explicit restricted source-year PUF CSV used only + by :func:`source_year_puf_adjusted_gross_income`. + donor_build_summary: Optional mutable receipt populated by shared donor + construction, including capital-gains and mortgage-quarantine + diagnostics. + + Returns: + The tax-unit-grain PUF donor used by the support-transfer stages. + + Raises: + ValueError: If the source-year path is absent or its records do not + align exactly to the processed PUF IDs and weights. + """ + + arrays = _read_processed_puf_arrays(Path(processed_puf_h5)) + if source_year_puf_csv is None: + raise ValueError( + "--puf-source-year-csv is required to align nonzero E19200 records " + "to the published TY2015 SOI AGI bands." + ) + adjusted_gross_income = source_year_puf_adjusted_gross_income( + Path(source_year_puf_csv), + processed_tax_unit_ids=arrays["tax_unit_id"], + processed_tax_unit_weights=arrays["household_weight"], + ) + return puf_tax_unit_donor_from_arrays( + arrays, + adjusted_gross_income=adjusted_gross_income, + donor_build_summary=donor_build_summary, + ) + + +def _read_processed_puf_arrays(path: Path) -> dict[str, np.ndarray]: + """Read root datasets without interpreting or rewriting stored arrays.""" + + import h5py + + with h5py.File(path, "r") as h5: + return {name: np.asarray(dataset) for name, dataset in h5.items()} diff --git a/packages/populace-build/tests/test_us_puf_donor_io.py b/packages/populace-build/tests/test_us_puf_donor_io.py new file mode 100644 index 00000000..a8f11683 --- /dev/null +++ b/packages/populace-build/tests/test_us_puf_donor_io.py @@ -0,0 +1,147 @@ +from __future__ import annotations + +from pathlib import Path + +import h5py +import numpy as np +import pandas as pd +import pytest + +from populace.build.us_runtime import load_puf_tax_unit_donor, puf_donor_io +from populace.build.us_runtime.puf_source_agi import ( + PUF_SOURCE_YEAR_AGI_REQUIRED_COLUMNS, +) + + +def _write_processed_arrays(path: Path, arrays: dict[str, np.ndarray]) -> None: + with h5py.File(path, "w") as h5: + for name, values in arrays.items(): + h5.create_dataset(name, data=values) + + +def test_loader_threads_source_alignment_and_summary( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + processed_path = tmp_path / "processed.h5" + source_path = tmp_path / "puf_2015.csv" + source_path.write_text("restricted source is read by the alignment seam") + arrays = { + "tax_unit_id": np.asarray([10, 20], dtype=np.int64), + "household_weight": np.asarray([1.0, 2.0]), + "untouched": np.asarray([3, 4], dtype=np.int16), + } + _write_processed_arrays(processed_path, arrays) + adjusted_gross_income = np.asarray([-5_000.0, 250_000.0]) + captured: dict[str, object] = {} + + def fake_agi( + actual_source_path: Path, + *, + processed_tax_unit_ids, + processed_tax_unit_weights, + ) -> np.ndarray: + captured["source_path"] = actual_source_path + captured["processed_tax_unit_ids"] = processed_tax_unit_ids + captured["processed_tax_unit_weights"] = processed_tax_unit_weights + return adjusted_gross_income + + expected = pd.DataFrame({"tax_unit_id": [10, 20]}) + + def fake_donor( + actual_arrays, + *, + adjusted_gross_income, + donor_build_summary, + ) -> pd.DataFrame: + captured["arrays"] = actual_arrays + captured["adjusted_gross_income"] = adjusted_gross_income + captured["donor_build_summary"] = donor_build_summary + donor_build_summary["mortgage_field_quarantine"] = {"screened_record_count": 2} + return expected + + monkeypatch.setattr( + puf_donor_io, + "source_year_puf_adjusted_gross_income", + fake_agi, + ) + monkeypatch.setattr( + puf_donor_io, + "puf_tax_unit_donor_from_arrays", + fake_donor, + ) + + summary: dict[str, object] = {} + actual = load_puf_tax_unit_donor( + processed_path, + source_path, + donor_build_summary=summary, + ) + + assert actual is expected + assert captured["source_path"] == source_path + np.testing.assert_array_equal( + captured["processed_tax_unit_ids"], + arrays["tax_unit_id"], + ) + np.testing.assert_array_equal( + captured["processed_tax_unit_weights"], + arrays["household_weight"], + ) + loaded_arrays = captured["arrays"] + assert isinstance(loaded_arrays, dict) + assert set(loaded_arrays) == set(arrays) + for name, values in arrays.items(): + np.testing.assert_array_equal(loaded_arrays[name], values) + assert captured["adjusted_gross_income"] is adjusted_gross_income + assert captured["donor_build_summary"] is summary + assert summary == {"mortgage_field_quarantine": {"screened_record_count": 2}} + + +def test_loader_refuses_missing_source_year_path(tmp_path: Path) -> None: + processed_path = tmp_path / "processed.h5" + _write_processed_arrays( + processed_path, + { + "tax_unit_id": np.asarray([1], dtype=np.int64), + "household_weight": np.asarray([1.0]), + }, + ) + + with pytest.raises( + ValueError, + match=( + "--puf-source-year-csv is required to align nonzero E19200 records " + "to the published TY2015 SOI AGI bands" + ), + ): + load_puf_tax_unit_donor(processed_path, None) + + +def test_loader_surfaces_source_year_recid_order_refusal(tmp_path: Path) -> None: + processed_path = tmp_path / "processed.h5" + _write_processed_arrays( + processed_path, + { + "tax_unit_id": np.asarray([2, 1, 1_000_000], dtype=np.int64), + "household_weight": np.asarray([1.0, 1.0, 1.0]), + }, + ) + source_path = tmp_path / "puf_2015.csv" + source = pd.DataFrame( + { + column: np.zeros(6, dtype=np.float64) + for column in PUF_SOURCE_YEAR_AGI_REQUIRED_COLUMNS + } + ) + source["RECID"] = [1, 2, 999_996, 999_997, 999_998, 999_999] + source["MARS"] = [1, 1, 0, 0, 0, 0] + source["S006"] = 100.0 + source["E00100"] = [10_000, 20_000, -1, 1, 10_000_000, 100_000_000] + source.to_csv(source_path, index=False) + + with pytest.raises( + ValueError, + match="Processed PUF regular RECID order does not match the TY2015 source", + ): + load_puf_tax_unit_donor(processed_path, source_path) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index d94245ab..b6ebcbb6 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -122,6 +122,7 @@ "pregnancy.py", "puf_aggregate_records.py", "puf_capital_gains_tail.py", + "puf_donor_io.py", # Bounded donor artifact I/O; no population treatment. "puf_e01000_reconciliation.py", "puf_interest_components.py", "puf_qrf_chain.py", From 13d84fd3d24dd8ae1be70159335bfb24449820ee Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 20:32:17 -0400 Subject: [PATCH 05/94] Add pool-safe seed and simulation stages --- PROGRESS.md | 10 + .../build/us_runtime/multispine_pool.py | 306 ++++++++++++++++-- .../src/populace/build/us_runtime/take_up.py | 70 +++- .../tests/test_us_multispine_pool.py | 148 +++++++-- 4 files changed, 473 insertions(+), 61 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 393d7b0d..30b39702 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -45,6 +45,16 @@ - Added small two-source tests for the ordered path, batched red agreement, receipt loss, clone-safe ID refusal, manifest counts, and default take-up inventory coverage. The new seam and the full spine-blindness guard pass. +- Added a public, bounded PUF donor loader that retains the existing + source-year E00100 alignment and processed-H5 refusal contracts. +- Made the TANF/EITC seed stage pool-safe: clone-stable IDs are a rowwise + fallback for draw keys, and assembled-pool cells already carrying values are + never overwritten. Other unresolved take-up defaults are filled only with + the live engine default and explicitly labeled as such; a missing + transfer-owned flag still fails closed. +- Added fixed-batch SSI materialization on an ephemeral, receipt-preserving + gate view. Formula-owned `ssi` is deliberately absent from the returned + input pool. ## Next diff --git a/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py b/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py index e0618799..4e1281fa 100644 --- a/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py +++ b/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py @@ -14,6 +14,10 @@ from collections.abc import Callable, Mapping from dataclasses import dataclass, field +from typing import Protocol + +import numpy as np +import pandas as pd from populace.build.gates import GateResult from populace.build.us_runtime.acs_transfer import ( @@ -28,16 +32,25 @@ spine_provenance_counts, validate_assembly_provenance, ) -from populace.build.us_runtime.take_up_contract import load_take_up_contract +from populace.build.us_runtime.take_up import with_us_take_up_inputs +from populace.build.us_runtime.take_up_contract import ( + TakeUpProgram, + load_take_up_contract, +) from populace.frame import Frame __all__ = [ "POOL_HOUSEHOLD_MASS_SHARES", "POOL_OPERATOR_ORDER", + "POOL_RANDOM_SEED", + "POOL_SIMULATION_HOUSEHOLD_BATCH_SIZE", + "POOL_TIME_PERIOD", "MultispinePoolResult", "PoolStageOutput", + "materialize_multispine_agreement_outputs", "pool_transfer_target_families", "run_multispine_pool_path", + "seed_multispine_pool_inputs", ] POOL_HOUSEHOLD_MASS_SHARES: Mapping[str, float] = { @@ -57,6 +70,26 @@ ) """The executable pool-build order, including the terminal QA evaluation.""" +POOL_RANDOM_SEED = 0 +"""Fixed seed shared by pool imputations and seeded input stages.""" + +POOL_TIME_PERIOD = 2024 +"""PolicyEngine period of the 2024 source pool.""" + +POOL_SIMULATION_HOUSEHOLD_BATCH_SIZE = 5_000 +"""Fixed household batch size for terminal formula-output evaluation.""" + + +class _PoolRulesEngine(Protocol): + def default_values(self, names: list[str]) -> Mapping[str, object]: ... + + def materialize( + self, + bundle: Frame, + variables: list[str], + period: int, + ) -> Mapping[str, np.ndarray]: ... + @dataclass(frozen=True) class PoolStageOutput: @@ -97,36 +130,267 @@ def simulation_ready(self) -> bool: def pool_transfer_target_families() -> TargetFamilies: - """Return the fixed pool transfer plan checked by the #581 gate. + """Return the fixed raw-preserving QRF transfer plan. - The existing ACS QRF declaration remains the base. Every take-up input in - the checked-in contract is added when that declaration does not already - own it, because the ACS peer has no measured take-up flags and the terminal - default agreement registry requires the complete take-up surface. + The #581 default agreement registry supplements this declaration with the + complete take-up inventory and formula-owned SSI. Take-up inputs not owned + by the declared QRF are handled together in the later seed stage, where + sourced TANF/EITC draws and explicitly disclosed engine defaults remain + distinguishable in the receipt. """ - plan = { + return { entity: {family: tuple(columns) for family, columns in families.items()} for entity, families in declared_acs_transfer_target_families().items() } - owned = { - (entity, column) - for entity, families in plan.items() + + +def seed_multispine_pool_inputs( + frame: Frame, + *, + engine: _PoolRulesEngine | None = None, +) -> PoolStageOutput: + """Seed sourced flags, then disclose and fill unresolved engine defaults. + + TANF and EITC use their checked-in administrative seed mechanisms over the + whole assembled pool. Existing non-null values survive byte-for-byte. + Other take-up owners are not fabricated here: any unresolved cells receive + the installed engine's declared default and the receipt names that fact, + the contract treatment, and its scope owner/follow-up evidence. + """ + + contract = load_take_up_contract() + before = _take_up_snapshots(frame, contract.programs) + seeded = with_us_take_up_inputs( + frame, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ) + _assert_take_up_values_preserved(before, seeded) + + rules_engine = engine + if rules_engine is None: + from populace.frame.adapters.policyengine_us import PolicyEngineUSEngine + + rules_engine = PolicyEngineUSEngine() + names = [program.variable for program in contract.programs] + defaults = dict(rules_engine.default_values(names)) + transfer_owned = { + column + for families in pool_transfer_target_families().values() for columns in families.values() for column in columns } - additions: dict[str, list[str]] = {} - for program in load_take_up_contract().programs: - key = (program.entity, program.variable) - if key in owned: + + tables = {entity: seeded.table(entity).copy() for entity in seeded.entities} + programs: dict[str, dict[str, object]] = {} + for program in contract.programs: + table = tables[program.entity] + if program.variable in table: + values = table[program.variable].copy() + missing = values.isna() + else: + values = pd.Series(pd.NA, index=table.index, dtype="boolean") + missing = pd.Series(True, index=table.index) + + if program.is_seeded: + if missing.any(): + raise ValueError( + f"Seeded take-up input {program.variable!r} still has " + f"{int(missing.sum())} missing row(s)." + ) + provenance_kind = "administrative_seed_or_preserved_input" + defaulted_rows = 0 + seeded_rows = int( + len(table) + if program.variable not in before + else before[program.variable][1].isna().sum() + ) + elif program.variable in transfer_owned: + if missing.any(): + raise ValueError( + f"Transfer-owned take-up input {program.variable!r} still " + f"has {int(missing.sum())} missing row(s); refusing to hide " + "an incomplete transfer behind an engine default." + ) + provenance_kind = "transferred_or_preserved_input" + defaulted_rows = 0 + seeded_rows = 0 + else: + if program.variable not in defaults: + raise ValueError( + "PolicyEngine exposes no input default for take-up variable " + f"{program.variable!r}." + ) + default = defaults[program.variable] + if default != program.default: + raise ValueError( + f"Take-up contract default for {program.variable!r} is " + f"{program.default!r}, but the installed engine reports " + f"{default!r}." + ) + values.loc[missing] = default + table[program.variable] = values.astype(bool) + provenance_kind = "preserved_input_or_disclosed_engine_default" + defaulted_rows = int(missing.sum()) + seeded_rows = 0 + + rate = program.rate + programs[program.variable] = { + "entity": program.entity, + "populace_treatment": program.populace_treatment, + "provenance_kind": provenance_kind, + "preserved_nonnull_rows": int(len(table) - defaulted_rows), + "seeded_rows": seeded_rows, + "defaulted_rows": defaulted_rows, + "engine_default": program.default, + "administrative_source": rate.get("source"), + "administrative_rate_status": rate.get("status"), + "scope_owner": program.raw.get("scope_owner"), + "followup": program.raw.get("followup"), + } + + result = Frame( + tables, + seeded.schema, + {entity: seeded.weights_for(entity) for entity in seeded.weighted_entities}, + seeded.strata, + mass_log=seeded.mass_log, + metadata=seeded.metadata, + ) + _assert_take_up_values_preserved(before, result) + return PoolStageOutput( + result, + { + "seed": POOL_RANDOM_SEED, + "time_period": POOL_TIME_PERIOD, + "programs": programs, + }, + ) + + +def materialize_multispine_agreement_outputs( + frame: Frame, + *, + engine: _PoolRulesEngine | None = None, +) -> PoolStageOutput: + """Materialize SSI in fixed household batches on an ephemeral gate view. + + The returned frame preserves the assembly receipt and adds ``person.ssi``. + The caller must gate this view and publish :attr:`MultispinePoolResult.frame` + instead; persisting ``ssi`` would pin a formula-owned output and mask + reforms. + """ + + if any("ssi" in frame.table(entity) for entity in frame.entities): + raise ValueError( + "Multispine agreement simulation refuses a persisted 'ssi' column; " + "SSI must remain formula-owned and gate-view-only." + ) + rules_engine = engine + if rules_engine is None: + from populace.frame.adapters.policyengine_us import PolicyEngineUSEngine + + rules_engine = PolicyEngineUSEngine() + + household_ids = frame.table("household")["household_id"].to_numpy() + person = frame.table("person") + membership = person["person_household_id"] + person_ids = person["person_id"] + if person_ids.duplicated().any(): + raise ValueError("Multispine SSI materialization requires unique person IDs.") + + values_by_person_id = pd.Series( + np.nan, + index=pd.Index(person_ids.to_numpy(), name="person_id"), + dtype=np.float64, + ) + batch_count = 0 + for low in range(0, len(household_ids), POOL_SIMULATION_HOUSEHOLD_BATCH_SIZE): + selected_households = household_ids[ + low : low + POOL_SIMULATION_HOUSEHOLD_BATCH_SIZE + ] + person_mask = membership.isin(selected_households).to_numpy() + selected = frame.select(person_mask) + materialized = np.asarray( + rules_engine.materialize( + selected, + ["ssi"], + POOL_TIME_PERIOD, + )["ssi"], + dtype=np.float64, + ) + selected_ids = selected.table("person")["person_id"].to_numpy() + if materialized.shape != (len(selected_ids),): + raise ValueError( + "Materialized SSI does not align with the selected person table." + ) + values_by_person_id.loc[selected_ids] = materialized + batch_count += 1 + + if values_by_person_id.isna().any(): + raise ValueError( + "Batched SSI materialization did not cover every person exactly once." + ) + tables = {entity: frame.table(entity).copy() for entity in frame.entities} + tables["person"]["ssi"] = values_by_person_id.reindex(person_ids).to_numpy() + evaluation = Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + return PoolStageOutput( + evaluation, + { + "formula_outputs": { + "ssi": { + "entity": "person", + "period": POOL_TIME_PERIOD, + "rows": int(len(person)), + } + }, + "household_batch_size": POOL_SIMULATION_HOUSEHOLD_BATCH_SIZE, + "batches": batch_count, + "persisted_to_pool": False, + }, + ) + + +def _take_up_snapshots( + frame: Frame, + programs: tuple[TakeUpProgram, ...], +) -> dict[str, tuple[str, pd.Series, pd.Series]]: + snapshots: dict[str, tuple[str, pd.Series, pd.Series]] = {} + for program in programs: + table = frame.table(program.entity) + if program.variable not in table: continue - additions.setdefault(program.entity, []).append(program.variable) - owned.add(key) - for entity, columns in additions.items(): - families = plan.setdefault(entity, {}) - existing = families.get("take_up", ()) - families["take_up"] = (*existing, *sorted(columns)) - return plan + values = table[program.variable].copy(deep=True) + snapshots[program.variable] = ( + program.entity, + values, + values.notna(), + ) + return snapshots + + +def _assert_take_up_values_preserved( + snapshots: Mapping[str, tuple[str, pd.Series, pd.Series]], + frame: Frame, +) -> None: + for variable, (entity, before, observed) in snapshots.items(): + after = frame.table(entity)[variable] + before_values = before.loc[observed].to_numpy(dtype=object) + after_values = after.loc[observed].to_numpy(dtype=object) + if not np.array_equal(before_values, after_values): + raise ValueError( + f"Pool take-up stage would overwrite non-null input " + f"{entity}.{variable}; measured/source-owned values must remain " + "untouched." + ) def run_multispine_pool_path( diff --git a/packages/populace-build/src/populace/build/us_runtime/take_up.py b/packages/populace-build/src/populace/build/us_runtime/take_up.py index f7c7f0a4..3c49c7b6 100644 --- a/packages/populace-build/src/populace/build/us_runtime/take_up.py +++ b/packages/populace-build/src/populace/build/us_runtime/take_up.py @@ -49,6 +49,9 @@ import pandas as pd from populace.build.gates import GateResult +from populace.build.us_runtime.support_provenance import ( + support_source_id_column, +) from populace.build.us_runtime.take_up_contract import ( TakeUpProgram, load_take_up_contract, @@ -107,7 +110,11 @@ def _person_membership_column(entity: str) -> str: def _stable_unit_draws( - units: pd.DataFrame, *, id_column: str, seed: int, variable: str + units: pd.DataFrame, + *, + id_column: str, + seed: int, + variable: str, ) -> np.ndarray: """Seeded uniform draws keyed by stable source identity per unit. @@ -116,16 +123,24 @@ def _stable_unit_draws( clones of one source unit — which share that identity — always draw together. Frames without source columns key on the entity id itself. """ + entity = id_column.removesuffix("_id") + source_id_column = support_source_id_column(entity) + keys = units[id_column].astype(str) + if source_id_column in units: + source_ids = units[source_id_column] + keys = keys.where(source_ids.isna(), source_ids.astype(str)) if set(_SOURCE_IDENTITY_COLUMNS) <= set(units.columns): - keys = ( + complete_identity = ( + units.loc[:, list(_SOURCE_IDENTITY_COLUMNS)].notna().all(axis=1) + ) + source_keys = ( units["source_year"].astype(str) + ":" + units["source_household_id"].astype(str) + ":" + units["source_person_id"].astype(str) ) - else: - keys = units[id_column].astype(str) + keys = keys.where(~complete_identity, source_keys) denominator = float(2**64) return np.asarray( [ @@ -156,6 +171,9 @@ def _units_with_source_identity(frame: Frame, entity: str) -> pd.DataFrame: id_column = _entity_id_column(entity) table = frame.table(entity) units = pd.DataFrame({id_column: table[id_column].to_numpy()}) + source_id = support_source_id_column(entity) + if source_id in table: + units[source_id] = table[source_id].to_numpy(copy=True) units["_weight"] = np.asarray( frame.resolve_weights(entity).values, dtype=np.float64 ) @@ -235,7 +253,10 @@ def _seed_program( id_column = _entity_id_column(entity) units = _units_with_source_identity(frame, entity) draws = _stable_unit_draws( - units, id_column=id_column, seed=seed, variable=program.variable + units, + id_column=id_column, + seed=seed, + variable=program.variable, ) if program.variable == "takes_up_eitc": @@ -292,11 +313,13 @@ def with_us_take_up_inputs( ) -> Frame: """Seed every ``seed`` take-up flag from the contract onto a US frame. - For each seeded program, assigns its ``takes_up_*`` flag across all units of - the flag's entity at the administrative rate (per-child for EITC). A frame - already carrying a non-constant column for a program is left untouched for - that program (idempotent); a missing or constant column is (re)computed — - the published all-True landmine is repaired, not trusted. + For each seeded program, assigns its ``takes_up_*`` flag at the + administrative rate (per-child for EITC). On an assembled pool, existing + non-null cells are measured/source-owned and remain untouched; only missing + cells receive seeded values. A complete assembled column is idempotent, + including a constant one, which its downstream signal gate may reject. + Legacy unassembled frames retain the prior behavior: a missing or constant + column is recomputed to repair the published all-True landmine. Args: frame: A US-schema frame carrying ``age`` on the person table and the @@ -319,14 +342,25 @@ def with_us_take_up_inputs( seeded = programs if programs is not None else seeded_take_up_programs() tables = {entity: frame.table(entity).copy() for entity in frame.entities} + assembled = all( + support_source_id_column(entity) in frame.table(entity) + for entity in frame.entities + ) changed = False for program in seeded: entity = program.entity table = tables[entity] - if program.variable in table.columns and _column_carries_signal( - table, program.variable - ): - continue + if program.variable in table.columns: + if assembled: + missing = table[program.variable].isna() + if not missing.any(): + continue + elif _column_carries_signal(table, program.variable): + continue + else: + missing = pd.Series(True, index=table.index) + else: + missing = pd.Series(True, index=table.index) column, _ = _seed_program(frame, program, seed=seed) id_column = _entity_id_column(entity) assigned = ( @@ -339,7 +373,13 @@ def with_us_take_up_inputs( f"US take-up stage output for {program.variable} does not cover " f"every {entity}." ) - table[program.variable] = assigned.to_numpy(dtype=bool) + if program.variable not in table: + table[program.variable] = assigned.to_numpy(dtype=bool) + else: + updated = table[program.variable].copy() + missing_positions = missing.to_numpy(dtype=bool) + updated.loc[missing] = assigned.to_numpy(dtype=bool)[missing_positions] + table[program.variable] = updated changed = True if not changed: diff --git a/packages/populace-build/tests/test_us_multispine_pool.py b/packages/populace-build/tests/test_us_multispine_pool.py index 5b68ccee..50610ff0 100644 --- a/packages/populace-build/tests/test_us_multispine_pool.py +++ b/packages/populace-build/tests/test_us_multispine_pool.py @@ -7,11 +7,16 @@ import pytest from populace.build.gates import GateResult +from populace.build.us_runtime.acs_transfer import ( + declared_acs_transfer_target_families, +) from populace.build.us_runtime.multispine_pool import ( POOL_OPERATOR_ORDER, PoolStageOutput, + materialize_multispine_agreement_outputs, pool_transfer_target_families, run_multispine_pool_path, + seed_multispine_pool_inputs, ) from populace.build.us_runtime.puf_support import ( PUF_SUPPORT_MAX_CLONE_SAFE_SOURCE_ID, @@ -20,6 +25,12 @@ SpineAgreementSpec, spine_agreement_gate, ) +from populace.build.us_runtime.spine_assembly import assemble_spines +from populace.build.us_runtime.support_provenance import ( + support_clone_index_column, + support_source_id_column, +) +from populace.build.us_runtime.take_up_contract import load_take_up_contract from populace.frame import US_SCHEMA, Frame, WeightKind, Weights @@ -248,28 +259,115 @@ def unreachable(_frame: Frame) -> PoolStageOutput: ) -def test_pool_transfer_plan_covers_every_take_up_without_duplicates() -> None: - families = pool_transfer_target_families() - ownership: dict[str, tuple[str, str]] = {} - for entity, by_family in families.items(): - for family, columns in by_family.items(): - for column in columns: - assert column not in ownership - ownership[column] = (entity, family) - - expected = { - "takes_up_snap_if_eligible", - "takes_up_tanf_if_eligible", - "takes_up_eitc", - "takes_up_medicaid_if_eligible", - "takes_up_chip_if_eligible", - "takes_up_basic_health_program_if_eligible", - "takes_up_medicare_if_eligible", - "takes_up_ssi_if_eligible", - "takes_up_dc_ptc", - "takes_up_head_start_if_eligible", - "takes_up_early_head_start_if_eligible", - "takes_up_housing_assistance_if_eligible", - "takes_up_aca_if_eligible", - } - assert expected <= set(ownership) +def test_pool_transfer_plan_is_the_fixed_declared_qrf_surface() -> None: + assert pool_transfer_target_families() == declared_acs_transfer_target_families() + + +class _FakeEngine: + def __init__(self) -> None: + self.materialized_person_ids: list[list[int]] = [] + + def default_values(self, names: list[str]) -> dict[str, object]: + programs = load_take_up_contract().program_map() + return {name: programs[name].default for name in names} + + def materialize( + self, + bundle: Frame, + variables: list[str], + period: int, + ) -> dict[str, np.ndarray]: + assert variables == ["ssi"] + assert period == 2024 + person = bundle.table("person") + self.materialized_person_ids.append(person["person_id"].astype(int).tolist()) + return {"ssi": person["age"].to_numpy(dtype=np.float64)} + + +def _assembled_cloned_with_partial_take_up() -> Frame: + asec = _source_frame() + tables = {entity: asec.table(entity).copy() for entity in asec.entities} + tables["spm_unit"]["takes_up_tanf_if_eligible"] = [True, False] + tables["spm_unit"]["takes_up_housing_assistance_if_eligible"] = [True, False] + tables["person"]["takes_up_medicare_if_eligible"] = [False, True] + asec = Frame( + tables, + asec.schema, + {"household": asec.weights_for("household")}, + asec.strata, + ) + acs = _source_frame() + tables = {entity: acs.table(entity).copy() for entity in acs.entities} + tables["spm_unit"]["takes_up_housing_assistance_if_eligible"] = [False, True] + acs = Frame( + tables, + acs.schema, + {"household": acs.weights_for("household")}, + acs.strata, + ) + assembled = assemble_spines( + {"asec": asec, "acs": acs}, + household_mass_shares={"asec": 0.5, "acs": 0.5}, + ) + from populace.build.us_runtime.puf_support import ( + clone_us_frame_for_puf_support, + ) + + return clone_us_frame_for_puf_support(assembled) + + +def test_pool_seed_stage_preserves_inputs_and_receipts_disclosed_defaults() -> None: + frame = _assembled_cloned_with_partial_take_up() + before_person = frame.table("person") + before_spm = frame.table("spm_unit") + engine = _FakeEngine() + + result = seed_multispine_pool_inputs(frame, engine=engine) + + after_person = result.frame.table("person") + after_spm = result.frame.table("spm_unit") + measured_person = before_person["takes_up_medicare_if_eligible"].notna() + measured_spm = before_spm["takes_up_tanf_if_eligible"].notna() + assert ( + after_person.loc[measured_person, "takes_up_medicare_if_eligible"].tolist() + == before_person.loc[measured_person, "takes_up_medicare_if_eligible"].tolist() + ) + assert ( + after_spm.loc[measured_spm, "takes_up_tanf_if_eligible"].tolist() + == before_spm.loc[measured_spm, "takes_up_tanf_if_eligible"].tolist() + ) + + contract = load_take_up_contract() + for program in contract.programs: + assert not result.frame.table(program.entity)[program.variable].isna().any() + tanf = result.receipt["programs"]["takes_up_tanf_if_eligible"] + assert tanf["provenance_kind"] == "administrative_seed_or_preserved_input" + medicare = result.receipt["programs"]["takes_up_medicare_if_eligible"] + assert medicare["provenance_kind"] == ( + "preserved_input_or_disclosed_engine_default" + ) + assert medicare["defaulted_rows"] == 4 + + spm = result.frame.table("spm_unit") + source_id = support_source_id_column("spm_unit") + clone_index = support_clone_index_column("spm_unit") + for _source, rows in spm.groupby(source_id): + assert set(rows[clone_index]) == {0, 1} + assert rows["takes_up_tanf_if_eligible"].nunique() == 1 + + +def test_simulated_ssi_lives_only_on_receipt_preserving_gate_view() -> None: + frame = seed_multispine_pool_inputs( + _assembled_cloned_with_partial_take_up(), + engine=_FakeEngine(), + ).frame + engine = _FakeEngine() + + result = materialize_multispine_agreement_outputs(frame, engine=engine) + + assert "ssi" not in frame.table("person") + assert "ssi" in result.frame.table("person") + assert result.frame.metadata == frame.metadata + assert result.receipt["persisted_to_pool"] is False + assert result.receipt["formula_outputs"]["ssi"]["rows"] == frame.n("person") + assert sum(map(len, engine.materialized_person_ids)) == frame.n("person") From 7f9426ef9f651fb939cc88cb08b09a3d9c59c168 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 20:34:27 -0400 Subject: [PATCH 06/94] Retire legacy ACS builder behind shared H5 I/O --- .../src/populace/build/us_runtime/h5_io.py | 222 ++++ .../test_us_acs_multispine_base_builder.py | 1087 ++------------- .../tests/test_us_spine_blindness.py | 1 + tools/build_us_acs_multispine_base.py | 1172 +---------------- 4 files changed, 399 insertions(+), 2083 deletions(-) create mode 100644 packages/populace-build/src/populace/build/us_runtime/h5_io.py diff --git a/packages/populace-build/src/populace/build/us_runtime/h5_io.py b/packages/populace-build/src/populace/build/us_runtime/h5_io.py new file mode 100644 index 00000000..c30edd01 --- /dev/null +++ b/packages/populace-build/src/populace/build/us_runtime/h5_io.py @@ -0,0 +1,222 @@ +"""PolicyEngine-compatible US H5 I/O for nullable build artifacts. + +The writer stores entity tables, household weights, the period, and a small +artifact metadata record. Fixed-format pandas tables preserve nullable +object columns without filling or coercing measured values. The companion +build manifest, rather than this consumer H5, owns stage receipts such as +``Frame.metadata`` and ``Frame.mass_log``. +""" + +from __future__ import annotations + +import json +import os +import uuid +import warnings +from pathlib import Path + +import numpy as np +import pandas as pd + +from populace.frame import Frame, WeightKind, Weights +from populace.frame.units import US_SCHEMA + +__all__ = [ + "LEGACY_NULLABLE_STAGING_ARTIFACT_KIND", + "load_legacy_calibrated_us_h5", + "write_nullable_us_h5", +] + +LEGACY_NULLABLE_STAGING_ARTIFACT_KIND = "nullable_precalibration_staging_h5" +_METADATA_KEY = "_populace_staging_metadata" +_TIME_PERIOD_KEY = "_time_period" + + +def load_legacy_calibrated_us_h5(path: str | Path) -> Frame: + """Load a legacy US single-year H5 as a calibrated-weight frame. + + Legacy PolicyEngine US artifacts do not expose typed weight provenance + through ``USSingleYearDataset``. This loader therefore preserves the + historical builder contract and labels their household weights + ``CALIBRATED``. It is not the loader for the new pre-calibration + multispine pool, whose importance-weight receipt lives in its manifest. + """ + + from policyengine_us.data import USSingleYearDataset + + dataset = USSingleYearDataset(file_path=str(Path(path))) + tables = { + "person": dataset.person, + "household": dataset.household.copy(), + "tax_unit": dataset.tax_unit, + "spm_unit": dataset.spm_unit, + "family": dataset.family, + "marital_unit": dataset.marital_unit, + } + household_weights = ( + tables["household"].pop("household_weight").to_numpy(dtype=np.float64) + ) + return Frame( + tables, + US_SCHEMA, + { + "household": Weights( + household_weights, + WeightKind.CALIBRATED, + ) + }, + ) + + +def write_nullable_us_h5( + frame: Frame, + path: str | Path, + *, + period: int, + artifact_kind: str, +) -> None: + """Atomically write and verify a nullable US single-year H5. + + The destination is replaced only after a temporary sibling has round-trip + verified every nonempty entity table, household weights, period metadata, + fixed-format storage, and the caller-declared ``artifact_kind``. A failed + write or verification leaves any existing destination bytes untouched. + """ + + if not isinstance(frame, Frame): + raise TypeError(f"frame must be a Frame, got {type(frame).__name__}.") + if not isinstance(artifact_kind, str) or not artifact_kind.strip(): + raise ValueError("artifact_kind must be a non-empty string.") + + output = Path(path) + output.parent.mkdir(parents=True, exist_ok=True) + temporary = output.with_name(f".{output.name}.{uuid.uuid4().hex}.tmp") + try: + _write_nullable_us_h5_file( + frame, + temporary, + period=int(period), + artifact_kind=artifact_kind, + ) + _verify_nullable_us_h5( + frame, + temporary, + period=int(period), + artifact_kind=artifact_kind, + ) + os.replace(temporary, output) + except BaseException: + temporary.unlink(missing_ok=True) + raise + + +def _write_nullable_us_h5_file( + frame: Frame, + path: Path, + *, + period: int, + artifact_kind: str, +) -> None: + with pd.HDFStore(path, mode="w") as store: + for entity in frame.entities: + table = _export_table(frame, entity) + if not len(table): + continue + # Fixed format preserves mixed bool/null object columns + # losslessly. Table format rejects them, which would force a + # fill or type rewrite. + with warnings.catch_warnings(): + warnings.simplefilter("ignore", pd.errors.PerformanceWarning) + store.put(entity, table, format="fixed") + store.put( + _TIME_PERIOD_KEY, + pd.Series([period]), + format="table", + ) + store.put( + _METADATA_KEY, + pd.Series( + [ + json.dumps( + _artifact_metadata(frame, artifact_kind=artifact_kind), + sort_keys=True, + ) + ] + ), + format="table", + ) + + +def _verify_nullable_us_h5( + frame: Frame, + path: Path, + *, + period: int, + artifact_kind: str, +) -> None: + with pd.HDFStore(path, mode="r") as store: + for entity in frame.entities: + expected = _export_table(frame, entity) + if not len(expected): + continue + try: + stored = store[entity] + except KeyError as exc: + raise RuntimeError( + f"Nullable US H5 round trip omitted entity {entity!r}." + ) from exc + try: + pd.testing.assert_frame_equal( + stored, + expected, + check_exact=True, + ) + except AssertionError as exc: + raise RuntimeError( + f"Nullable US H5 round trip changed entity {entity!r}: {exc}" + ) from exc + if store.get_storer(entity).is_table: + raise RuntimeError( + f"Nullable US H5 stored entity {entity!r} in table format." + ) + + stored_period = store[_TIME_PERIOD_KEY] + if stored_period.tolist() != [period]: + raise RuntimeError( + "Nullable US H5 round trip changed the time period: " + f"expected {period}, got {stored_period.tolist()}." + ) + raw_metadata = store[_METADATA_KEY] + if len(raw_metadata) != 1: + raise RuntimeError( + "Nullable US H5 must carry exactly one artifact metadata row." + ) + try: + stored_metadata = json.loads(str(raw_metadata.iloc[0])) + except (TypeError, ValueError) as exc: + raise RuntimeError( + "Nullable US H5 artifact metadata is not valid JSON." + ) from exc + expected_metadata = _artifact_metadata(frame, artifact_kind=artifact_kind) + if stored_metadata != expected_metadata: + raise RuntimeError( + "Nullable US H5 round trip changed artifact metadata: " + f"expected {expected_metadata}, got {stored_metadata}." + ) + + +def _export_table(frame: Frame, entity: str) -> pd.DataFrame: + table = frame.table(entity) + if entity != "household": + return table + household = table.copy() + household["household_weight"] = frame.weights_for("household").values + return household + + +def _artifact_metadata(frame: Frame, *, artifact_kind: str) -> dict[str, str]: + return { + "artifact_kind": artifact_kind, + "entity_hdf_format": "fixed_nullable", + "household_weight_kind": frame.weights_for("household").kind.value, + } diff --git a/packages/populace-build/tests/test_us_acs_multispine_base_builder.py b/packages/populace-build/tests/test_us_acs_multispine_base_builder.py index e7f5207c..87f905cd 100644 --- a/packages/populace-build/tests/test_us_acs_multispine_base_builder.py +++ b/packages/populace-build/tests/test_us_acs_multispine_base_builder.py @@ -1,31 +1,40 @@ +"""Compatibility tests for the retired ACS builder shim and shared US H5 I/O.""" + from __future__ import annotations -import hashlib import importlib.util +import inspect import json +import subprocess +import sys from pathlib import Path -from types import SimpleNamespace +from types import ModuleType import numpy as np import pandas as pd import pytest -from populace.build import FitWeightRecord -from populace.build.us_runtime.acs_sources import ( - AcsSourceArtifact, - AcsSourceManifest, +from populace.build.us_runtime import h5_io +from populace.build.us_runtime.h5_io import ( + LEGACY_NULLABLE_STAGING_ARTIFACT_KIND, + load_legacy_calibrated_us_h5, + write_nullable_us_h5, ) -from populace.build.us_runtime.base_pool import spine_column -from populace.build.us_runtime.puf_support import support_channel_column from populace.frame import US_SCHEMA, Frame, WeightKind, Weights -def _load_builder_module(): - root = Path(__file__).resolve().parents[3] - path = root / "tools" / "build_us_acs_multispine_base.py" +def _shim_path() -> Path: + return ( + Path(__file__).resolve().parents[3] + / "tools" + / ("build_us_acs_multispine_base.py") + ) + + +def _load_shim_module(): spec = importlib.util.spec_from_file_location( "build_us_acs_multispine_base", - path, + _shim_path(), ) module = importlib.util.module_from_spec(spec) assert spec.loader is not None @@ -33,11 +42,7 @@ def _load_builder_module(): return module -def _frame( - *, - benefit_participation: bool = True, - spines: tuple[str, str] | None = None, -) -> Frame: +def _frame(*, weight_kind: WeightKind = WeightKind.IMPORTANCE) -> Frame: person = pd.DataFrame( { "person_id": [1, 2], @@ -47,984 +52,178 @@ def _frame( "person_family_id": [1, 2], "person_marital_unit_id": [1, 2], "age": [35, 67], - "is_female": [False, True], + "is_snap_abawd_discretionary_exempt": pd.Series( + [True, np.nan], + dtype=object, + ), } ) - if benefit_participation: - person["takes_up_snap_if_eligible"] = [True, False] tables = { "person": person, - "household": pd.DataFrame({"household_id": [1, 2], "state_fips": [6, 36]}), + "household": pd.DataFrame( + { + "household_id": [1, 2], + "state_fips": [6, 36], + } + ), "tax_unit": pd.DataFrame({"tax_unit_id": [1, 2]}), "spm_unit": pd.DataFrame({"spm_unit_id": [1, 2]}), "family": pd.DataFrame({"family_id": [1, 2]}), "marital_unit": pd.DataFrame({"marital_unit_id": [1, 2]}), } - if spines is not None: - for entity, table in tables.items(): - table[spine_column(entity)] = list(spines) return Frame( tables, US_SCHEMA, { "household": Weights( np.asarray([40.0, 60.0]), - WeightKind.CALIBRATED, + weight_kind, ) }, ) -def _manifest() -> AcsSourceManifest: - source_directory = ( - "https://www2.census.gov/programs-surveys/acs/data/pums/2024/1-Year/" - ) - return AcsSourceManifest( - version=1, - spine="acs_2024_1yr", - vintage=2024, - verified_on="2026-07-10", - source_directory=source_directory, - artifacts=( - AcsSourceArtifact( - role="household", - filename="csv_hus.zip", - url=f"{source_directory}csv_hus.zip", - sha256="1" * 64, - size_bytes=123, - ), - AcsSourceArtifact( - role="person", - filename="csv_pus.zip", - url=f"{source_directory}csv_pus.zip", - sha256="2" * 64, - size_bytes=456, - ), - ), - ) - - -def test_parser_exposes_production_defaults_and_transfer_controls() -> None: - builder = _load_builder_module() +def _stored_metadata(path: Path) -> dict[str, object]: + with pd.HDFStore(path, mode="r") as store: + return json.loads(store["_populace_staging_metadata"].iloc[0]) - args = builder._parse_args( - [ - "--base-h5", - "dense.h5", - "--out-h5", - "combined.h5", - ] - ) - assert args.inputs_dir == builder.DEFAULT_INPUTS_DIR - assert args.inputs_dir.is_absolute() - assert args.inputs_dir.parts[-2:] == ("inputs", "acs_2024_1yr") - assert args.puma_ladder == builder.DEFAULT_PUMA_LADDER - assert args.puma_ladder.is_absolute() - assert args.n_estimators == 32 - assert args.max_targets_per_fit == 8 - assert args.period == 2024 - assert args.acs_share == 0.5 - assert args.seed == 0 - assert args.geography_seed == 0 - assert args.donor_channel == builder.ACS_DONOR_CHANNEL_AUTO - - custom = builder._parse_args( - [ - "--base-h5", - "dense.h5", - "--out-h5", - "combined.h5", - "--chunksize", - "1234", - "--acs-share", - "0.3", - "--seed", - "17", - "--geography-seed", - "23", - "--puma-ladder", - "ladder.npz", - "--n-estimators", - "9", - "--max-targets-per-fit", - "3", - "--donor-channel", - "benefit_support", - ] - ) - assert custom.chunksize == 1234 - assert custom.acs_share == 0.3 - assert custom.seed == 17 - assert custom.geography_seed == 23 - assert custom.puma_ladder == Path("ladder.npz") - assert custom.n_estimators == 9 - assert custom.max_targets_per_fit == 3 - assert custom.donor_channel == "benefit_support" - - -def test_main_wires_verified_sources_transfer_audit_export_and_summary( - monkeypatch: pytest.MonkeyPatch, +def test_nullable_writer_round_trips_fixed_tables_and_caller_artifact_kind( tmp_path: Path, ) -> None: - builder = _load_builder_module() - manifest = _manifest() - base = _frame() - combined = _frame(spines=("asec_puf", "acs_2024_1yr")) - combined.table("household")["puma"] = ["0100100", "0200100"] - combined.table("household")["congressional_district_geoid"] = [101, 200] - combined.table("household")["county_fips"] = ["01001", "02020"] - combined.table("household")["TYPEHUGQ"] = [1.0, 3.0] - base_h5 = tmp_path / "dense.h5" - base_h5.write_bytes(b"dense-base") - manifest_path = tmp_path / "acs_sources.json" - manifest_path.write_text("{}", encoding="utf-8") - puma_ladder_path = tmp_path / "us_puma_ladder_2020.npz" - puma_ladder_path.write_bytes(b"puma-ladder") - puma_ladder = builder.UsPumaLadder( - puma=np.asarray([100_100], dtype=np.int64), - puma_population=np.asarray([100.0]), - cd_overlap_puma=np.asarray([100_100], dtype=np.int64), - cd_overlap_cd=np.asarray([101], dtype=np.int64), - cd_overlap_population=np.asarray([100.0]), - county_overlap_puma=np.asarray([100_100], dtype=np.int64), - county_overlap_county=np.asarray([1_001], dtype=np.int32), - county_overlap_population=np.asarray([100.0]), - tract_overlap_puma=np.asarray([100_100], dtype=np.int64), - tract_overlap_tract=np.asarray([1_001_000_100], dtype=np.int64), - tract_overlap_population=np.asarray([100.0]), - metadata={ - "schema_version": 1, - "kind": "us_puma_ladder", - "puma_vintage": "2020_puma", - "sampling_basis": "population", - "layers": { - "congressional_district": {"vintage": "119th_congress"}, - "county": {"vintage": "2020_census"}, - "tract": {"vintage": "2020_census"}, - }, - }, - ) - inputs_dir = tmp_path / "inputs" - household_zip = inputs_dir / "csv_hus.zip" - person_zip = inputs_dir / "csv_pus.zip" - source = builder.AcsPumsSource(household_zip, person_zip) - output_h5 = tmp_path / "combined.h5" - summary_path = tmp_path / "combined.summary.json" - captured: dict[str, object] = {} - transfer_plan = { - "person": { - "benefit_participation": ("takes_up_snap_if_eligible",), - } - } - - def fake_load_manifest(path): - captured["manifest_path"] = path - return manifest - - def fake_fetch(cache_dir, *, manifest): - captured["fetch"] = (cache_dir, manifest) - return source - - def fake_build(actual_base, actual_source, **kwargs): - captured["build"] = (actual_base, actual_source, kwargs) - return builder.AcsMultispineResult( - frame=combined, - fit_records=(FitWeightRecord("acs_transfer:person:benefits", "design"),), - provenance={ - "enabled": True, - "deferred_inputs": ["block_geoid", "tract_geoid"], - "geography_ladder": { - "applied": True, - "household_rows": 2, - "ladder_pumas": 1, - "layer_vintages": puma_ladder.layer_vintages, - "sampling_basis": "population", - "seed": 19, - "resolved_model_inputs": [ - "congressional_district_geoid", - "county_fips", - ], - "unresolved_sub_puma_inputs": [ - "block_geoid", - "tract_geoid", - ], - }, - "imputed_inputs": [ - { - "column": "takes_up_snap_if_eligible", - "family": "benefit_participation", - "unmodeled_recipient_rows": 0, - } - ], - }, - ) - - def fake_write(frame, path, *, period): - captured["write"] = (frame, path, period) - path.write_bytes(b"combined-output") - - def fake_load_puma_ladder(path): - captured["puma_ladder_path"] = path - return puma_ladder + pytest.importorskip("tables") + frame = _frame() + output = tmp_path / "pool.h5" - monkeypatch.setattr(builder, "_load_base_frame", lambda path: base) - monkeypatch.setattr( - builder, - "_require_dense_donor_coverage", - lambda frame, **kwargs: None, + write_nullable_us_h5( + frame, + output, + period=2024, + artifact_kind="nullable_multispine_pool_h5", ) - monkeypatch.setattr( - builder, - "declared_acs_transfer_target_families", - lambda: transfer_plan, - ) - monkeypatch.setattr( - builder.acs_sources, - "load_acs_source_manifest", - fake_load_manifest, - ) - monkeypatch.setattr( - builder.acs_sources, - "fetch_acs_pums_sources", - fake_fetch, - ) - monkeypatch.setattr(builder, "build_optional_acs_multispine", fake_build) - monkeypatch.setattr( - builder, - "load_us_puma_ladder", - fake_load_puma_ladder, - ) - monkeypatch.setattr(builder, "_engine_input_null_audit", lambda frame: []) - monkeypatch.setattr( - builder, - "_preflight_staging_export", - lambda frame: 123_456, - ) - monkeypatch.setattr(builder, "_write_dataset", fake_write) - exit_code = builder.main( - [ - "--base-h5", - str(base_h5), - "--out-h5", - str(output_h5), - "--summary", - str(summary_path), - "--source-manifest", - str(manifest_path), - "--inputs-dir", - str(inputs_dir), - "--puma-ladder", - str(puma_ladder_path), - "--max-households", - "7", - "--chunksize", - "2000", - "--acs-share", - "0.4", - "--seed", - "11", - "--geography-seed", - "19", - ] - ) - - assert exit_code == 0 - assert captured["manifest_path"] == manifest_path - assert captured["puma_ladder_path"] == puma_ladder_path - assert captured["fetch"] == (inputs_dir, manifest) - actual_base, actual_source, build_options = captured["build"] - assert actual_base is base - assert actual_source.max_households == 7 - assert build_options.pop("puma_ladder") is puma_ladder - assert build_options == { - "chunksize": 2000, - "acs_share": 0.4, - "target_families": transfer_plan, - "donor_channel": builder.ACS_DONOR_CHANNEL_AUTO, - "seed": 11, - "n_estimators": 32, - "max_targets_per_fit": 8, - "geography_seed": 19, - } - assert captured["write"] == (combined, output_h5, 2024) - - summary = json.loads(summary_path.read_text(encoding="utf-8")) - assert summary["artifact_kind"] == "nullable_precalibration_staging_h5" - assert summary["calibration_applied"] is False - assert summary["simulation_ready"] is False - assert summary["simulation_ready_except_calibration"] is True - assert summary["simulation_readiness_blockers"] == ["calibration_not_applied"] - assert [item["id"] for item in summary["reviewed_limitations"]] == [ - "acs_group_quarters_housing_universe", - "native_acs_source_universe_blanks", - "sub_puma_geographic_precision", - ] - assert summary["reviewed_limitations"][2]["unavailable_exact_geography"] == [ - "block_geoid", - "tract_geoid", - ] - assert summary["reviewed_engine_input_nulls"] == [] - assert "pending_engine_input_nulls" not in summary - assert summary["staging_export_peak_estimate_bytes"] == 123_456 - assert summary["geography_ladder"] == { - "path": str(puma_ladder_path.resolve()), - "sha256": hashlib.sha256(b"puma-ladder").hexdigest(), - "pumas": 1, - "layer_vintages": { - "congressional_district": "119th_congress", - "county": "2020_census", - "puma": "2020_puma", - "tract": "2020_census", - }, - "seed": 19, - "assignment": summary["orchestration"]["provenance"]["geography_ladder"], - } - assert summary["acs_sources"]["manifest"] == str(manifest_path.resolve()) - assert ( - summary["acs_sources"]["manifest_sha256"] == hashlib.sha256(b"{}").hexdigest() - ) - assert summary["acs_sources"]["artifacts"] == [ - { - "role": "household", - "filename": "csv_hus.zip", - "url": manifest.artifacts[0].url, - "sha256": "1" * 64, - "size_bytes": 123, - "local_path": str(household_zip.resolve()), - }, - { - "role": "person", - "filename": "csv_pus.zip", - "url": manifest.artifacts[1].url, - "sha256": "2" * 64, - "size_bytes": 456, - "local_path": str(person_zip.resolve()), - }, - ] - assert summary["weights_audit"]["details"]["resolved_weight_kinds"] == { - "acs_transfer:person:benefits": "design" - } - assert summary["rows"]["combined"] == {entity: 2 for entity in US_SCHEMA.entities} - assert summary["household_weight_totals"] == { - "base": 100.0, - "combined": 100.0, - } - assert summary["spine_totals"] == { - "acs_2024_1yr": { - "rows": {entity: 1 for entity in US_SCHEMA.entities}, - "household_weight_total": 60.0, - }, - "asec_puf": { - "rows": {entity: 1 for entity in US_SCHEMA.entities}, - "household_weight_total": 40.0, - }, + with pd.HDFStore(output, mode="r") as store: + assert store.get_storer("person").is_table is False + assert store["person"]["is_snap_abawd_discretionary_exempt"].tolist()[0] is True + assert pd.isna(store["person"]["is_snap_abawd_discretionary_exempt"].iloc[1]) + assert store["household"]["household_weight"].tolist() == [40.0, 60.0] + assert store["_time_period"].tolist() == [2024] + assert _stored_metadata(output) == { + "artifact_kind": "nullable_multispine_pool_h5", + "entity_hdf_format": "fixed_nullable", + "household_weight_kind": "importance", } - assert summary["output"]["sha256"] == hashlib.sha256(b"combined-output").hexdigest() -def test_weights_audit_failure_aborts_before_export() -> None: - builder = _load_builder_module() - result = builder.AcsMultispineResult( - frame=_frame(), - fit_records=(FitWeightRecord("acs_transfer:person:benefits", "none"),), - ) - - with pytest.raises(SystemExit) as exc: - builder._audit_fits(result) - - assert "Weights audit failed" in str(exc.value) - assert "acs_transfer:person:benefits" in str(exc.value) - assert "unweighted" in str(exc.value) - - -def test_default_transfer_fails_closed_before_fetch_without_benefit_participation( +def test_nullable_writer_keeps_existing_bytes_when_verification_fails( monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, ) -> None: - builder = _load_builder_module() - monkeypatch.setattr( - builder.acs_sources, - "load_acs_source_manifest", - lambda path: _manifest(), - ) - monkeypatch.setattr( - builder, - "_load_base_frame", - lambda path: _frame(benefit_participation=False), - ) - monkeypatch.setattr(builder, "_sha256", lambda path: "0" * 64) - - def must_not_fetch(*_args, **_kwargs): - raise AssertionError("source fetch must not run for an incomplete donor") - - monkeypatch.setattr( - builder.acs_sources, - "fetch_acs_pums_sources", - must_not_fetch, - ) - - with pytest.raises(SystemExit) as exc: - builder.main( - [ - "--base-h5", - "incomplete.h5", - "--out-h5", - "combined.h5", - ] - ) - - assert "no takes_up_*" in str(exc.value) - assert "must run after the benefit input-family stages" in str(exc.value) - - -def test_empty_transfer_audit_is_not_treated_as_success() -> None: - builder = _load_builder_module() - - with pytest.raises(SystemExit) as exc: - builder._audit_fits( - SimpleNamespace( - fit_records=(), - ) - ) - - assert "produced no fit records" in str(exc.value) - + pytest.importorskip("tables") + output = tmp_path / "existing.h5" + output.write_bytes(b"existing-artifact") -class _DefaultsEngine: - _defaults = { - "age": 0, - "alimony_income": 0.0, - "employment_income_before_lsr": 0.0, - "has_esi": False, - "is_female": False, - "state_fips": 0, - "takes_up_snap_if_eligible": True, - "weekly_hours_worked_before_lsr": 40.0, - } - - def default_values(self, names): - return {name: self._defaults[name] for name in names if name in self._defaults} - - -def test_dense_donor_ignores_default_runtime_column_not_consumed_by_transfer() -> None: - builder = _load_builder_module() - base = _frame() - base.person["weekly_hours_worked_before_lsr"] = [40.0, 40.0] - - builder._require_dense_donor_coverage( - base, - engine=_DefaultsEngine(), - donor_channel=None, - target_families={ - "person": { - "benefit_participation": ("takes_up_snap_if_eligible",), - } - }, - ) + def fail_verification(*_args, **_kwargs): + raise RuntimeError("injected verification failure") + monkeypatch.setattr(h5_io, "_verify_nullable_us_h5", fail_verification) -def test_dense_donor_missing_transfer_consumed_column_fails_hard() -> None: - builder = _load_builder_module() - - with pytest.raises(SystemExit) as exc: - builder._require_dense_donor_coverage( + with pytest.raises(RuntimeError, match="injected verification failure"): + write_nullable_us_h5( _frame(), - engine=_DefaultsEngine(), - donor_channel=None, - target_families={"person": {"health": ("has_esi",)}}, + output, + period=2024, + artifact_kind="fixture", ) - message = str(exc.value) - assert "hard ACS transfer-consumption gate" in message - assert "person.has_esi" in message - assert "transfer-consumed column is absent" in message - - -def test_dense_donor_default_transfer_consumed_column_fails_hard() -> None: - builder = _load_builder_module() - base = _frame() - base.person["has_esi"] = [False, False] - - with pytest.raises(SystemExit) as exc: - builder._require_dense_donor_coverage( - base, - engine=_DefaultsEngine(), - donor_channel=None, - target_families={"person": {"health": ("has_esi",)}}, - ) - - message = str(exc.value) - assert "hard ACS transfer-consumption gate" in message - assert "person.has_esi" in message - assert "every observed donor value equals the engine default" in message - - -def test_dense_donor_default_consumed_optional_feature_fails_hard() -> None: - builder = _load_builder_module() - base = _frame() - base.person["employment_income_before_lsr"] = [0.0, 0.0] - - with pytest.raises(SystemExit) as exc: - builder._require_dense_donor_coverage( - base, - engine=_DefaultsEngine(), - donor_channel=None, - target_families={ - "person": { - "benefit_participation": ("takes_up_snap_if_eligible",), - } - }, - ) - - message = str(exc.value) - assert "person.employment_income_before_lsr" in message - assert "every observed donor value equals the engine default" in message - - -def test_signal_bearing_release_exclusion_is_irrelevant_to_donor_gate( - capsys: pytest.CaptureFixture[str], -) -> None: - builder = _load_builder_module() - base = _frame() - base.person["alimony_income"] = [0.0, 100.0] - - builder._require_dense_donor_coverage( - base, - engine=_DefaultsEngine(), - donor_channel=None, - target_families={ - "person": { - "benefit_participation": ("takes_up_snap_if_eligible",), - } - }, - ) - - captured = capsys.readouterr() - assert captured.out == "" - assert captured.err == "" - - -def test_dense_donor_coverage_is_checked_on_resolved_puf_channel( - monkeypatch: pytest.MonkeyPatch, -) -> None: - builder = _load_builder_module() - base = _frame() - channel_column = support_channel_column("person") - for entity in base.entities: - base.table(entity)[support_channel_column(entity)] = [ - "asec", - "puf_tax_detail", - ] - captured: dict[str, Frame] = {} - - def fake_requirements(frame, target_families): - captured["frame"] = frame - return {"person": ("age",)} - - monkeypatch.setattr( - builder, - "acs_transfer_donor_requirements", - fake_requirements, - ) - - builder._require_dense_donor_coverage( - base, - engine=_DefaultsEngine(), - target_families={"person": {"test": ("age",)}}, - ) - - selected = captured["frame"] - assert selected.n("person") == 1 - assert selected.person[channel_column].tolist() == ["puf_tax_detail"] + assert output.read_bytes() == b"existing-artifact" + assert list(tmp_path.glob(f".{output.name}.*.tmp")) == [] -def test_partial_donor_channel_metadata_fails_before_coverage_gate( - monkeypatch: pytest.MonkeyPatch, +def test_shim_preserves_legacy_write_signature_and_default( + tmp_path: Path, ) -> None: - builder = _load_builder_module() - base = _frame() - base.table("household")[support_channel_column("household")] = [ - "asec", - "puf_tax_detail", + pytest.importorskip("tables") + shim = _load_shim_module() + signature = inspect.signature(shim._write_dataset) + + assert list(signature.parameters) == [ + "frame", + "path", + "period", + "artifact_kind", ] - called = False - - def fake_requirements(frame, target_families): - nonlocal called - called = True - return {"person": ("age",)} - - monkeypatch.setattr( - builder, - "acs_transfer_donor_requirements", - fake_requirements, + assert signature.parameters["period"].kind is inspect.Parameter.KEYWORD_ONLY + assert signature.parameters["artifact_kind"].kind is ( + inspect.Parameter.KEYWORD_ONLY ) - - with pytest.raises(SystemExit, match="partial support metadata"): - builder._require_dense_donor_coverage( - base, - engine=_DefaultsEngine(), - target_families={"person": {"test": ("age",)}}, - ) - - assert not called - - -def test_default_transfer_cannot_report_success_without_benefit_imputation() -> None: - builder = _load_builder_module() - result = builder.AcsMultispineResult( - frame=_frame(), - provenance={ - "imputed_inputs": [ - { - "column": "qualified_dividend_income", - "family": "puf_tax_itemization", - } - ] - }, + assert signature.parameters["artifact_kind"].default == ( + LEGACY_NULLABLE_STAGING_ARTIFACT_KIND ) - with pytest.raises(SystemExit) as exc: - builder._require_benefit_participation_transfer(result) - - assert "no takes_up_* benefit-participation input" in str(exc.value) + output = tmp_path / "legacy-staging.h5" + shim._write_dataset(_frame(), output, period=2024) - -def test_default_transfer_must_register_every_planned_input( - monkeypatch: pytest.MonkeyPatch, -) -> None: - builder = _load_builder_module() - result = builder.AcsMultispineResult( - frame=_frame(), - provenance={ - "imputed_inputs": [ - { - "column": "takes_up_snap_if_eligible", - "family": "benefit_participation", - } - ] - }, + assert _stored_metadata(output)["artifact_kind"] == ( + LEGACY_NULLABLE_STAGING_ARTIFACT_KIND ) - monkeypatch.setattr( - builder, - "default_acs_transfer_target_families", - lambda donor: { - "person": { - "benefit_participation": ("takes_up_snap_if_eligible",), - "model_required": ("has_esi",), - } - }, - ) - - with pytest.raises(SystemExit) as exc: - builder._require_default_transfer_coverage(result, _frame()) - - assert "omitted donor-observed model input" in str(exc.value) - assert "has_esi" in str(exc.value) - - -def test_staging_h5_round_trips_base_only_nullable_boolean(tmp_path: Path) -> None: - pytest.importorskip("tables") # pandas HDF backend - builder = _load_builder_module() - frame = _frame(spines=("asec_puf", "acs_2024_1yr")) - frame.table("person")["is_snap_abawd_discretionary_exempt"] = pd.Series( - [True, np.nan], dtype=object - ) - output = tmp_path / "nullable-bool-staging.h5" - - builder._write_dataset(frame, output, period=2024) - - stored = pd.read_hdf(output, key="person") - assert stored.loc[0, "is_snap_abawd_discretionary_exempt"] is True - assert pd.isna(stored.loc[1, "is_snap_abawd_discretionary_exempt"]) - - -def test_engine_input_null_audit_includes_float_bool_and_spine_counts() -> None: - builder = _load_builder_module() - frame = _frame(spines=("asec_puf", "acs_2024_1yr")) - frame.person["age"] = [35.0, np.nan] - frame.person["takes_up_snap_if_eligible"] = pd.Series([True, np.nan], dtype=object) - - class FakeEngine: - def variables(self): - return ["age", "takes_up_snap_if_eligible"] - - def variable_metadata(self, name): - return SimpleNamespace( - dtype={ - "age": "float", - "takes_up_snap_if_eligible": "bool", - }[name] - ) - - audit = builder._engine_input_null_audit(frame, FakeEngine()) - - assert audit == [ - { - "entity": "person", - "column": "age", - "dtype": "float", - "missing_rows": 1, - "rows": 2, - "missing_rows_by_spine": {"acs_2024_1yr": 1}, - }, - { - "entity": "person", - "column": "takes_up_snap_if_eligible", - "dtype": "bool", - "missing_rows": 1, - "rows": 2, - "missing_rows_by_spine": {"acs_2024_1yr": 1}, - }, - ] - - -def test_nullable_staging_writer_round_trips_group_quarters_blanks( - tmp_path: Path, -) -> None: - pytest.importorskip("tables") # pandas HDF backend - builder = _load_builder_module() - frame = _frame(spines=("asec_puf", "acs_2024_1yr")) - frame.person["real_estate_taxes"] = [1_000.0, np.nan] - frame.person["pre_subsidy_rent"] = [1_200.0, np.nan] - frame.table("household")["TYPEHUGQ"] = [1.0, 3.0] - frame.table("household")["tenure_type"] = ["RENTED", np.nan] - frame.table("spm_unit")["spm_unit_tenure_type"] = ["RENTER", np.nan] - output = tmp_path / "staging.h5" - - builder._write_dataset(frame, output, period=2024) - - with pd.HDFStore(output, mode="r") as store: - assert store["person"]["real_estate_taxes"].isna().sum() == 1 - assert store["person"]["pre_subsidy_rent"].isna().sum() == 1 - assert store["household"]["tenure_type"].isna().sum() == 1 - assert store["spm_unit"]["spm_unit_tenure_type"].isna().sum() == 1 - assert store["household"]["household_weight"].tolist() == [40.0, 60.0] - metadata = json.loads(store["_populace_staging_metadata"].iloc[0]) - assert metadata == { - "artifact_kind": "nullable_precalibration_staging_h5", - "entity_hdf_format": "fixed_nullable", - "household_weight_kind": "calibrated", - } - - -def test_staging_export_preflight_enforces_decimal_30gb_limit() -> None: - builder = _load_builder_module() - frame = _frame() - - estimate = builder._preflight_staging_export(frame) - assert estimate < builder.DEFAULT_STAGING_EXPORT_PEAK_LIMIT_BYTES - with pytest.raises(MemoryError, match="above the 0.00 GB limit"): - builder._preflight_staging_export(frame, max_peak_bytes=1) -def test_registered_transfer_with_unmodeled_required_rows_fails( +def test_legacy_loader_and_shim_keep_calibrated_weight_contract( monkeypatch: pytest.MonkeyPatch, ) -> None: - builder = _load_builder_module() - frame = _frame(spines=("asec_puf", "acs_2024_1yr")) - frame.person["has_esi"] = pd.Series([True, np.nan], dtype=object) - result = builder.AcsMultispineResult( - frame=frame, - provenance={ - "imputed_inputs": [ - { - "column": "has_esi", - "family": "model_required_boolean", - "unmodeled_recipient_rows": 1, - } - ] - }, - ) - monkeypatch.setattr( - builder, - "default_acs_transfer_target_families", - lambda donor: {"person": {"model_required_boolean": ("has_esi",)}}, - ) - - with pytest.raises(SystemExit, match="left 1 unmodeled row"): - builder._require_default_transfer_coverage(result, _frame()) - - -def test_group_quarters_rent_gap_is_explicit_structural_pending( - monkeypatch: pytest.MonkeyPatch, -) -> None: - builder = _load_builder_module() - frame = _frame(spines=("asec_puf", "acs_2024_1yr")) - frame.person["pre_subsidy_rent"] = [1_000.0, np.nan] - frame.table("household")["TYPEHUGQ"] = [np.nan, 3.0] - result = builder.AcsMultispineResult( - frame=frame, - provenance={ - "imputed_inputs": [ - { - "column": "pre_subsidy_rent", - "family": "housing", - "unmodeled_recipient_rows": 1, - } - ] - }, - ) - monkeypatch.setattr( - builder, - "default_acs_transfer_target_families", - lambda donor: {"person": {"housing": ("pre_subsidy_rent",)}}, - ) - - coverage = builder._require_default_transfer_coverage(result, _frame()) - - assert coverage["structural_pending"] == [ - { - "column": "pre_subsidy_rent", - "entity": "person", - "rows": 1, - "reason": ( - "ACS group-quarters rows are outside the housing-tenure universe" - ), - } - ] - - -def test_reviewed_limitations_close_gq_and_sub_puma_gaps() -> None: - builder = _load_builder_module() - frame = _frame(spines=("asec_puf", "acs_2024_1yr")) - frame.table("household")["TYPEHUGQ"] = [np.nan, 3.0] - result = builder.AcsMultispineResult( - frame=frame, - provenance={ - "geography_ladder": { - "applied": True, - "seed": 29, - "layer_vintages": { - "puma": "2020_puma", - "congressional_district": "119th_congress", - "county": "2020_census", - "tract": "2020_census", - }, - "unresolved_sub_puma_inputs": ["block_geoid", "tract_geoid"], - } - }, - ) - transfer_coverage = { - "structural_pending": [ - { - "column": "pre_subsidy_rent", - "entity": "person", - "rows": 1, - "reason": ( - "ACS group-quarters rows are outside the housing-tenure universe" - ), - } - ] - } - input_null_audit = [ - { - "entity": "person", - "column": "pre_subsidy_rent", - "missing_rows_by_spine": {"acs_2024_1yr": 1}, - }, - { - "entity": "person", - "column": "employment_income_before_lsr", - "missing_rows_by_spine": {"acs_2024_1yr": 1}, - }, - ] - - limitations = builder._reviewed_limitations( - result, - transfer_coverage=transfer_coverage, - input_null_audit=input_null_audit, - ) - - gq, native, geography = limitations - assert gq["status"] == "reviewed_structural_absence" - assert gq["affected_rows"] == {"household": 1, "person": 1, "spm_unit": 1} - assert gq["engine_input_nulls"] == [input_null_audit[0]] - assert gq["transfer_evidence"] == transfer_coverage["structural_pending"] - assert native["engine_input_nulls_excluding_group_quarters_housing"] == [ - input_null_audit[1] - ] - assert geography["status"] == "reviewed_probabilistic_assignment" - assert geography["unavailable_exact_geography"] == [ - "block_geoid", - "tract_geoid", - ] - assert geography["assignment_seed"] == 29 - assert all(item["calibration_blocker"] is False for item in limitations) - - -def test_donor_release_identity_verifies_sha_and_pins_provenance( + source = _frame(weight_kind=WeightKind.DESIGN) + captured: dict[str, str] = {} + + class FakeDataset: + def __init__(self, *, file_path: str) -> None: + captured["file_path"] = file_path + for entity in source.entities: + table = source.table(entity).copy() + if entity == "household": + table["household_weight"] = [7.0, 11.0] + setattr(self, entity, table) + + package = ModuleType("policyengine_us") + package.__path__ = [] # type: ignore[attr-defined] + data = ModuleType("policyengine_us.data") + data.USSingleYearDataset = FakeDataset # type: ignore[attr-defined] + package.data = data # type: ignore[attr-defined] + monkeypatch.setitem(sys.modules, "policyengine_us", package) + monkeypatch.setitem(sys.modules, "policyengine_us.data", data) + + path = Path("legacy.h5") + direct = load_legacy_calibrated_us_h5(path) + via_shim = _load_shim_module()._load_base_frame(path) + + assert captured["file_path"] == str(path) + for loaded in (direct, via_shim): + assert loaded.weights_for("household").kind is WeightKind.CALIBRATED + assert loaded.weights_for("household").values.tolist() == [7.0, 11.0] + assert "household_weight" not in loaded.table("household") + + +def test_retired_cli_exits_nonzero_with_pool_builder_migration_command( tmp_path: Path, ) -> None: - module = _load_builder_module() - sha = "a" * 64 - manifest = tmp_path / "release_manifest.json" - manifest.write_text( - json.dumps( - { - "dataset_role": "national_default", - "is_default": True, - "build": {"build_id": "populace-us-2024-buildo-sparse-x"}, - "artifacts": { - "populace_us_2024": { - "kind": "microdata", - "path": "populace_us_2024.h5", - "repo_id": "policyengine/populace-us", - "revision": "populace-us-2024-buildo-sparse-x", - "sha256": sha, - }, - "calibration_diagnostics": { - "kind": "diagnostics", - "path": "calibration_diagnostics.json", - "sha256": "b" * 64, - }, - }, - } - ) - ) - - identity = module._donor_release_identity(manifest, sha) - - assert identity == { - "manifest_path": str(manifest.resolve()), - "artifact": "populace_us_2024", - "release_id": "populace-us-2024-buildo-sparse-x", - "revision": "populace-us-2024-buildo-sparse-x", - "repo_id": "policyengine/populace-us", - "sha256": sha, - "dataset_role": "national_default", - "is_default": True, - } - assert module._donor_release_identity(None, sha) is None + output = tmp_path / "must-not-exist.h5" - -def test_donor_release_identity_rejects_sha_mismatch(tmp_path: Path) -> None: - module = _load_builder_module() - manifest = tmp_path / "release_manifest.json" - manifest.write_text( - json.dumps( - { - "artifacts": { - "populace_us_2024": { - "kind": "microdata", - "sha256": "a" * 64, - } - } - } - ) - ) - - with pytest.raises(SystemExit, match="does not match its release manifest"): - module._donor_release_identity(manifest, "c" * 64) - - -def test_donor_release_identity_requires_one_microdata_artifact( - tmp_path: Path, -) -> None: - module = _load_builder_module() - manifest = tmp_path / "release_manifest.json" - manifest.write_text(json.dumps({"artifacts": {}})) - - with pytest.raises(SystemExit, match="exactly one microdata artifact"): - module._donor_release_identity(manifest, "a" * 64) + result = subprocess.run( + [ + sys.executable, + str(_shim_path()), + "--base-h5", + "legacy.h5", + "--out-h5", + str(output), + ], + check=False, + capture_output=True, + text=True, + ) + + assert result.returncode == 2 + assert result.stdout == "" + assert "legacy command cannot be translated" in result.stderr + assert "uv run tools/build_us_multispine_pool.py --help" in result.stderr + assert not output.exists() diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index b6ebcbb6..eec8157c 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -111,6 +111,7 @@ "fiscal_targets.py", "geography_ladder.py", "hours_worked.py", + "h5_io.py", # US artifact I/O; no population treatment. "immigration.py", "input_mass.py", "l0_refit_export.py", diff --git a/tools/build_us_acs_multispine_base.py b/tools/build_us_acs_multispine_base.py index 2b1a1157..e3529e26 100644 --- a/tools/build_us_acs_multispine_base.py +++ b/tools/build_us_acs_multispine_base.py @@ -1,993 +1,40 @@ -"""Build a nullable ASEC-by-PUF plus ACS multispine staging artifact. +"""Deprecated compatibility shim for the retired late-ACS builder. -This tool starts from the dense, input-complete ASEC-by-PUF H5 produced by -the US input-family pipeline. It acquires the byte-pinned ACS PUMS archives, -loads and maps the ACS spine, transfers model input leaves from the dense -donor, assigns the PUMA-anchored state/CD/county geography ladder, audits every -fit's resolved typed weight kind, and writes the combined base. Calibration is -deliberately downstream. +The executable build path moved to ``tools/build_us_multispine_pool.py``. +This module keeps only the two H5 helpers imported by the legacy ACS +local-release tool. It never runs or translates the retired late-assembly +pipeline. """ from __future__ import annotations -import argparse -import gc -import hashlib -import json -import os -import warnings -from dataclasses import replace +import sys from pathlib import Path -from typing import Any -import numpy as np -import pandas as pd - -from populace.build import ( - FitWeightRecord, - default_valued_columns_gate, - weights_audit_gate, -) -from populace.build.us_runtime import acs_sources -from populace.build.us_runtime.acs_multispine import ( - AcsMultispineResult, - build_optional_acs_multispine, -) -from populace.build.us_runtime.acs_pums import ( - ACS_2024_1YR_SPINE, - DEFAULT_CHUNKSIZE, - AcsPumsSource, -) -from populace.build.us_runtime.acs_transfer import ( - ACS_DONOR_CHANNEL_AUTO, - DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, - TargetFamilies, - acs_derived_transfer_expectations, - acs_transfer_donor_requirements, - declared_acs_transfer_target_families, - default_acs_transfer_target_families, - resolve_acs_donor_channel, -) -from populace.build.us_runtime.base_pool import spine_column -from populace.build.us_runtime.puma_ladder import ( - UsPumaLadder, - load_us_puma_ladder, -) -from populace.frame import Frame, WeightKind, Weights -from populace.frame.units import US_SCHEMA - -PERIOD = 2024 -_REPOSITORY_ROOT = Path(__file__).resolve().parents[1] -DEFAULT_INPUTS_DIR = _REPOSITORY_ROOT / "inputs" / "acs_2024_1yr" -DEFAULT_PUMA_LADDER = _REPOSITORY_ROOT / "build" / "us" / "us_puma_ladder_2020.npz" -DEFAULT_N_ESTIMATORS = 32 -DEFAULT_STAGING_EXPORT_PEAK_LIMIT_BYTES = int( - os.environ.get("POPULACE_STAGING_EXPORT_PEAK_LIMIT_BYTES", 30_000_000_000) -) -_STAGING_EXPORT_FIXED_OVERHEAD_BYTES = 512 * 1024**2 -_PACKAGED_MANIFEST_REFERENCE = ( - "package:populace.build.us_runtime/acs_2024_1yr_sources.json" +from populace.build.us_runtime.h5_io import ( + LEGACY_NULLABLE_STAGING_ARTIFACT_KIND, + load_legacy_calibrated_us_h5, + write_nullable_us_h5, ) +from populace.frame import Frame +__all__ = ["_load_base_frame", "_write_dataset", "main"] -def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: - parser = argparse.ArgumentParser( - description=( - "Append the pinned ACS 2024 1-year PUMS spine to an already-built " - "dense ASEC-by-PUF base. The nullable result has reviewed source-" - "universe limitations and is simulation-ready except for the " - "downstream calibration solve." - ) - ) - parser.add_argument("--base-h5", required=True, type=Path) - parser.add_argument("--out-h5", required=True, type=Path) - parser.add_argument( - "--summary", - type=Path, - help=( - "JSON build summary path. Defaults to OUT-H5 with a .summary.json suffix." - ), - ) - parser.add_argument( - "--source-manifest", - type=Path, - help="Strict ACS source manifest override; defaults to the packaged pin.", - ) - parser.add_argument( - "--inputs-dir", - default=DEFAULT_INPUTS_DIR, - type=Path, - help="Hash-verified ACS archive cache (default: inputs/acs_2024_1yr).", - ) - parser.add_argument( - "--puma-ladder", - default=DEFAULT_PUMA_LADDER, - type=Path, - help=( - "Validated national PUMA-ladder NPZ (default: " - "build/us/us_puma_ladder_2020.npz)." - ), - ) - parser.add_argument( - "--max-households", - type=_positive_int, - help="Optional deterministic smoke limit applied after ACS household sort.", - ) - parser.add_argument("--period", default=PERIOD, type=_positive_int) - parser.add_argument("--chunksize", default=DEFAULT_CHUNKSIZE, type=_positive_int) - parser.add_argument("--acs-share", default=0.5, type=_open_unit_interval) - parser.add_argument("--seed", default=0, type=_nonnegative_int) - parser.add_argument( - "--geography-seed", - default=0, - type=_nonnegative_int, - help="Deterministic PUMA/CD/county assignment seed (default: 0).", - ) - parser.add_argument( - "--n-estimators", - default=DEFAULT_N_ESTIMATORS, - type=_positive_int, - ) - parser.add_argument( - "--max-targets-per-fit", - default=DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, - type=_positive_int, - help="Maximum chained QRF targets retained at once (default: 8).", - ) - parser.add_argument( - "--donor-channel", - default=ACS_DONOR_CHANNEL_AUTO, - help=( - "ASEC-by-PUF support channel used by transfer. The default selects " - "the appropriate support channel for each input family." - ), - ) - parser.add_argument( - "--donor-release-manifest", - type=Path, - help=( - "release_manifest.json of the published release BASE-H5 came " - "from. When given, the donor H5's sha256 must match the " - "manifest's root microdata artifact, and the donor release " - "identity is pinned into the staging summary." - ), - ) - return parser.parse_args(argv) - - -def _donor_release_identity( - manifest_path: Path | None, - base_sha256: str, -) -> dict[str, object] | None: - """Verify the donor H5 against its release manifest and pin its identity.""" +_MIGRATION_MESSAGE = """\ +tools/build_us_acs_multispine_base.py is retired. - if manifest_path is None: - return None - try: - manifest = json.loads(manifest_path.read_text()) - except (OSError, ValueError) as exc: - raise SystemExit( - f"Cannot read donor release manifest {manifest_path}: {exc}." - ) from exc - artifacts = manifest.get("artifacts") - if not isinstance(artifacts, dict): - raise SystemExit( - f"Donor release manifest {manifest_path} carries no artifacts map." - ) - microdata = [ - (name, entry) - for name, entry in artifacts.items() - if isinstance(entry, dict) and entry.get("kind") == "microdata" - ] - if len(microdata) != 1: - raise SystemExit( - "Donor release manifest must carry exactly one microdata " - f"artifact; found {sorted(name for name, _ in microdata)}." - ) - name, entry = microdata[0] - if entry.get("sha256") != base_sha256: - raise SystemExit( - "Donor H5 sha256 does not match its release manifest: base-h5 " - f"is {base_sha256} but {manifest_path} pins " - f"{entry.get('sha256')!r} for {name!r}. Refusing to transfer " - "from an unverified donor." - ) - build = manifest.get("build") if isinstance(manifest.get("build"), dict) else {} - return { - "manifest_path": str(manifest_path.resolve()), - "artifact": name, - "release_id": build.get("build_id"), - "revision": entry.get("revision"), - "repo_id": entry.get("repo_id"), - "sha256": entry.get("sha256"), - "dataset_role": manifest.get("dataset_role"), - "is_default": manifest.get("is_default"), - } +Its --base-h5 input was already post-clone, so the legacy command cannot be translated +without violating the required assemble -> clone ordering. - -def main(argv: list[str] | None = None) -> int: - args = _parse_args(argv) - summary_path = args.summary or args.out_h5.with_suffix(".summary.json") - _validate_artifact_paths(args, summary_path=summary_path) - - manifest = acs_sources.load_acs_source_manifest(args.source_manifest) - manifest_file = _manifest_file(args.source_manifest) - manifest_sha256 = _sha256(manifest_file) - base_sha256 = _sha256(args.base_h5) - donor_release = _donor_release_identity(args.donor_release_manifest, base_sha256) - base = _load_base_frame(args.base_h5) - _require_benefit_participation_inputs(base) - transfer_plan = declared_acs_transfer_target_families() - _require_dense_donor_coverage( - base, - donor_channel=args.donor_channel, - target_families=transfer_plan, - ) - base_rows = _row_counts(base) - base_mass = float(base.weights_for("household").total) - puma_ladder_sha256 = _sha256(args.puma_ladder) - puma_ladder = load_us_puma_ladder(args.puma_ladder) - - source = acs_sources.fetch_acs_pums_sources( - args.inputs_dir, - manifest=manifest, - ) - if args.max_households is not None: - source = replace(source, max_households=args.max_households) - - result = build_optional_acs_multispine( - base, - source, - chunksize=args.chunksize, - acs_share=args.acs_share, - target_families=transfer_plan, - donor_channel=args.donor_channel, - seed=args.seed, - n_estimators=args.n_estimators, - max_targets_per_fit=args.max_targets_per_fit, - puma_ladder=puma_ladder, - geography_seed=args.geography_seed, - ) - _require_puma_ladder_assignment(result) - _require_benefit_participation_transfer(result) - transfer_coverage = _require_default_transfer_coverage( - result, - base, - target_families=transfer_plan, - ) - weights_audit = _audit_fits(result) - - # The pooled frame owns its assembled blocks. Release the dense donor - # before HDF serialization so export cannot retain both full spines plus - # writer scratch at once. - del base - gc.collect() - input_null_audit = _engine_input_null_audit(result.frame) - gc.collect() - - args.out_h5.parent.mkdir(parents=True, exist_ok=True) - staging_export_peak_bytes = _preflight_staging_export(result.frame) - _write_dataset(result.frame, args.out_h5, period=args.period) - - summary = _build_summary( - args=args, - summary_path=summary_path, - manifest=manifest, - source=source, - base_rows=base_rows, - base_mass=base_mass, - base_sha256=base_sha256, - donor_release=donor_release, - manifest_sha256=manifest_sha256, - result=result, - weights_audit=weights_audit, - transfer_coverage=transfer_coverage, - input_null_audit=input_null_audit, - staging_export_peak_bytes=staging_export_peak_bytes, - puma_ladder=puma_ladder, - puma_ladder_sha256=puma_ladder_sha256, - ) - summary_path.parent.mkdir(parents=True, exist_ok=True) - rendered = json.dumps(summary, indent=2, sort_keys=True, allow_nan=False) + "\n" - summary_path.write_text(rendered, encoding="utf-8") - print(rendered, end="") - return 0 - - -def _audit_fits(result: AcsMultispineResult) -> dict[str, object]: - records = result.fit_records - if not records: - raise SystemExit( - "ACS input transfer produced no fit records. The dense donor must " - "carry transferable tax-detail and benefit-participation inputs." - ) - non_typed = [ - type(record).__name__ - for record in records - if not isinstance(record, FitWeightRecord) - ] - if non_typed: - raise TypeError( - "ACS input transfer returned non-FitWeightRecord audit evidence: " - f"{non_typed}." - ) - gate = weights_audit_gate(records) - if not gate.passed: - raise SystemExit("Weights audit failed:\n " + "\n ".join(gate.failures)) - return { - "passed": gate.passed, - "failures": list(gate.failures), - "details": dict(gate.details), - } - - -def _require_benefit_participation_inputs(base: Frame) -> None: - participation = sorted( - column - for entity in base.entities - for column in base.table(entity).columns - if column.startswith("takes_up_") - ) - if not participation: - raise SystemExit( - "Dense ASEC-by-PUF donor has no takes_up_* benefit-participation " - "inputs. This tool must run after the benefit input-family stages." - ) - - -def _require_dense_donor_coverage( - base: Frame, - engine: Any | None = None, - *, - donor_channel: str | None = ACS_DONOR_CHANNEL_AUTO, - target_families: TargetFamilies | None = None, -) -> None: - """Fail unless every column consumed by the QRF plan has donor signal.""" - - if engine is None: - from populace.frame.adapters.policyengine_us import PolicyEngineUSEngine - - engine = PolicyEngineUSEngine() - selected, resolved_channel = _coverage_donor_channel(base, donor_channel) - plan = target_families or declared_acs_transfer_target_families() - try: - requirements = acs_transfer_donor_requirements(selected, plan) - except ValueError as exc: - raise SystemExit( - "Dense ASEC-by-PUF donor has an invalid ACS transfer feature " - f"surface; selected channel={resolved_channel!r}: {exc}." - ) from exc - - failures: list[str] = [] - present_values: dict[str, Any] = {} - owners: dict[str, str] = {} - for entity, columns in requirements.items(): - if entity not in selected.entities: - failures.extend( - f"{entity}.{column}: transfer-consumed column is absent " - "because the donor entity is missing." - for column in columns - ) - continue - table = selected.table(entity) - for column in columns: - if column not in table.columns: - actual_owner = _column_owner(selected, column) - location = ( - f"; found on entity {actual_owner!r}" - if actual_owner is not None - else "" - ) - failures.append( - f"{entity}.{column}: transfer-consumed column is absent{location}." - ) - continue - values = table[column].to_numpy() - if not pd.notna(values).any(): - failures.append( - f"{entity}.{column}: transfer-consumed column has no " - "observed donor values." - ) - continue - present_values[column] = values - owners[column] = entity - - defaults = engine.default_values(sorted(present_values)) - default_gate = default_valued_columns_gate(present_values, defaults) - default_valued = default_gate.details["default_valued_columns"] - for column, default in sorted(default_valued.items()): - failures.append( - f"{owners[column]}.{column}: every observed donor value equals " - f"the engine default ({default!r}); QRF transfer requires usable " - "target/predictor signal." - ) - - if failures: - raise SystemExit( - "Dense ASEC-by-PUF donor failed the hard ACS transfer-consumption " - f"gate; selected channel={resolved_channel!r}:\n " + "\n ".join(failures) - ) - - -def _column_owner(frame: Frame, column: str) -> str | None: - try: - return frame.column_entity(column) - except ValueError: - return None - - -def _coverage_donor_channel( - base: Frame, - requested: str | None, -) -> tuple[Frame, str | None]: - try: - return resolve_acs_donor_channel(base, requested) - except ValueError as exc: - raise SystemExit(f"Invalid ACS transfer donor support metadata: {exc}") from exc - - -def _require_benefit_participation_transfer(result: AcsMultispineResult) -> None: - imputed = result.provenance.get("imputed_inputs", []) - transferred = isinstance(imputed, list) and any( - isinstance(item, dict) - and isinstance(item.get("family"), str) - and item["family"].split("__batch_", 1)[0] == "benefit_participation" - and isinstance(item.get("column"), str) - and item["column"].startswith("takes_up_") - for item in imputed - ) - if not transferred: - raise SystemExit( - "ACS default transfer produced no takes_up_* benefit-participation " - "input. Refusing to report a complete multispine base." - ) - - -def _require_puma_ladder_assignment(result: AcsMultispineResult) -> None: - """Fail unless the enabled multispine resolved launch geography inputs.""" - - raw = result.provenance.get("geography_ladder") - if not isinstance(raw, dict) or raw.get("applied") is not True: - raise SystemExit( - "ACS multispine did not apply the required PUMA geography ladder." - ) - household = result.frame.table("household") - required = ("puma", "congressional_district_geoid", "county_fips") - missing = [column for column in required if column not in household] - if missing: - raise SystemExit( - f"PUMA geography assignment omitted household column(s): {missing}." - ) - nulls = { - column: int(household[column].isna().sum()) - for column in required - if household[column].isna().any() - } - if nulls: - raise SystemExit( - f"PUMA geography assignment left null household values: {nulls}." - ) - deferred = result.provenance.get("deferred_inputs", []) - still_deferred = sorted( - {"congressional_district_geoid", "county_fips"}.intersection(deferred) - if isinstance(deferred, list) - else {"congressional_district_geoid", "county_fips"} - ) - if still_deferred: - raise SystemExit( - "PUMA geography assignment still reports resolved input(s) as " - f"deferred: {still_deferred}." - ) - - -def _require_default_transfer_coverage( - result: AcsMultispineResult, - donor: Frame, - *, - target_families: TargetFamilies | None = None, -) -> dict[str, object]: - """Prove every planned target is present and complete on its ACS universe.""" - - expected: dict[str, str] = {} - plan = target_families or default_acs_transfer_target_families(donor) - for entity, entity_families in plan.items(): - for targets in entity_families.values(): - for target in targets: - expected[target] = entity - # Deterministically derived columns are as load-bearing as fitted ones: - # a plan carrying the CGD parents owes the derived memo leg too. - expected.update(acs_derived_transfer_expectations(plan)) - raw_imputed = result.provenance.get("imputed_inputs", []) - if not isinstance(raw_imputed, list): - raise SystemExit("ACS imputed-input provenance must be a JSON list.") - entries = { - item["column"]: item - for item in raw_imputed - if isinstance(item, dict) and isinstance(item.get("column"), str) - } - if len(entries) != len( - [item for item in raw_imputed if isinstance(item, dict) and "column" in item] - ): - raise SystemExit("ACS imputed-input provenance contains duplicate columns.") - missing = sorted(set(expected) - set(entries)) - if missing: - raise SystemExit( - "ACS default transfer omitted donor-observed model input(s): " - f"{missing}. Refusing to report a complete multispine base." - ) - - structural_pending: list[dict[str, object]] = [] - for column, entity in sorted(expected.items()): - table = result.frame.table(entity) - if column not in table: - raise SystemExit( - f"ACS transfer registered {column!r} but the combined {entity!r} " - "table does not contain it." - ) - tag = spine_column(entity) - if tag not in table: - raise SystemExit( - f"Combined staging frame lacks ACS spine tag {tag!r} on {entity!r}." - ) - acs_mask = table[tag].eq(ACS_2024_1YR_SPINE) - if not acs_mask.any(): - raise SystemExit(f"Combined staging frame has no ACS rows on {entity!r}.") - missing_mask = table[column].isna() & acs_mask - missing_rows = int(missing_mask.sum()) - raw_unmodeled = entries[column].get("unmodeled_recipient_rows", 0) - if type(raw_unmodeled) is not int or raw_unmodeled < 0: - raise SystemExit( - f"ACS imputation provenance for {column!r} has invalid " - f"unmodeled_recipient_rows={raw_unmodeled!r}." - ) - if missing_rows != raw_unmodeled: - raise SystemExit( - f"ACS transfer provenance for {column!r} reports " - f"{raw_unmodeled} unmodeled row(s), but the combined ACS spine " - f"contains {missing_rows} missing row(s)." - ) - if missing_rows == 0: - continue - if column != "pre_subsidy_rent" or entity != "person": - raise SystemExit( - f"ACS default transfer left {missing_rows} unmodeled row(s) in " - f"required input {column!r}." - ) - gq_mask = _acs_group_quarters_person_mask(result.frame) - if not missing_mask.equals(gq_mask): - raise SystemExit( - "ACS pre_subsidy_rent may remain absent only for native " - "group-quarters rows." - ) - structural_pending.append( - { - "column": column, - "entity": entity, - "rows": missing_rows, - "reason": "ACS group-quarters rows are outside the housing-tenure universe", - } - ) - - return { - "expected_inputs": sorted(expected), - "registered_inputs": sorted(entries), - "structural_pending": structural_pending, - } - - -def _acs_group_quarters_person_mask(frame: Frame) -> pd.Series: - household = frame.table("household") - person = frame.table("person") - required = {"household_id", "TYPEHUGQ", spine_column("household")} - missing = sorted(required - set(household.columns)) - if missing: - raise SystemExit( - "Cannot validate ACS group-quarters transfer gaps; household table " - f"lacks {missing}." - ) - kind = pd.to_numeric(household["TYPEHUGQ"], errors="coerce") - gq_households = household.loc[ - household[spine_column("household")].eq(ACS_2024_1YR_SPINE) & kind.isin([2, 3]), - "household_id", - ] - return person[spine_column("person")].eq(ACS_2024_1YR_SPINE) & person[ - "person_household_id" - ].isin(gq_households) - - -def _reviewed_limitations( - result: AcsMultispineResult, - *, - transfer_coverage: dict[str, object], - input_null_audit: list[dict[str, object]], -) -> list[dict[str, object]]: - """Document accepted source-universe and geographic precision limits.""" - - geography = result.provenance["geography_ladder"] - if not isinstance(geography, dict): # guarded by _require_puma_ladder_assignment - raise TypeError("geography_ladder provenance must be a mapping.") - - acs_nulls = [ - entry - for entry in input_null_audit - if isinstance(entry.get("missing_rows_by_spine"), dict) - and int(entry["missing_rows_by_spine"].get(ACS_2024_1YR_SPINE, 0)) > 0 - ] - gq_counts = _acs_group_quarters_counts(result.frame) - gq_entity_by_column = { - "pre_subsidy_rent": "person", - "spm_unit_tenure_type": "spm_unit", - "tenure_type": "household", - } - - def is_exact_gq_null(entry: dict[str, object]) -> bool: - column = entry.get("column") - entity = gq_entity_by_column.get(column) - missing_by_spine = entry.get("missing_rows_by_spine") - return ( - entity is not None - and entry.get("entity") == entity - and isinstance(missing_by_spine, dict) - and int(missing_by_spine.get(ACS_2024_1YR_SPINE, 0)) == gq_counts[entity] - ) - - gq_nulls = [entry for entry in acs_nulls if is_exact_gq_null(entry)] - other_native_nulls = [entry for entry in acs_nulls if not is_exact_gq_null(entry)] - structural_pending = transfer_coverage.get("structural_pending", []) - if not isinstance(structural_pending, list): - raise TypeError("transfer_coverage.structural_pending must be a list.") - - return [ - { - "id": "acs_group_quarters_housing_universe", - "status": "reviewed_structural_absence", - "affected_spine": ACS_2024_1YR_SPINE, - "affected_rows": gq_counts, - "affected_columns": { - "household": [ - "tenure_type", - "acs_monthly_contract_rent", - "acs_monthly_gross_rent", - "acs_annual_property_tax", - ], - "spm_unit": ["spm_unit_tenure_type"], - "person": ["pre_subsidy_rent", "real_estate_taxes"], - }, - "reason": ( - "ACS PUMS housing-unit tenure, rent, and property-tax fields " - "are outside the TYPEHUGQ 2/3 group-quarters universe." - ), - "treatment": ( - "Preserve those values as structural nulls; filling them with " - "zero or donor housing values would synthesize an unobserved " - "housing unit." - ), - "engine_input_nulls": gq_nulls, - "transfer_evidence": structural_pending, - "calibration_blocker": False, - }, - { - "id": "native_acs_source_universe_blanks", - "status": "reviewed_source_missingness", - "affected_spine": ACS_2024_1YR_SPINE, - "reason": ( - "Native ACS inputs retain official blank universes; the ACS " - "mapping contract forbids inventing zeros or component splits." - ), - "treatment": ( - "Preserve nullable source semantics after transferring every " - "donor-observed model-required input with an eligible fit." - ), - "engine_input_nulls_excluding_group_quarters_housing": (other_native_nulls), - "calibration_blocker": False, - }, - { - "id": "sub_puma_geographic_precision", - "status": "reviewed_probabilistic_assignment", - "observed_geography": { - "acs_2024_1yr": ["state_fips", "puma"], - "asec_puf": ( - [ - "state_fips", - "tract_geoid", - "congressional_district_geoid", - "county_fips", - ] - if geography.get("donor_geography") == "preserved_assigned" - else ["state_fips"] - ), - }, - "assigned_geography": [ - "puma", - "congressional_district_geoid", - "county_fips", - ], - "donor_geography": geography.get("donor_geography"), - "unavailable_exact_geography": list( - geography.get( - "unresolved_sub_puma_inputs", - ["block_geoid", "tract_geoid"], - ) - ), - "unavailable_exact_geography_scope": ( - "acs_2024_1yr" - if geography.get("donor_geography") == "preserved_assigned" - else "acs_2024_1yr,asec_puf" - ), - "reason": ( - "ACS PUMS identifies residence only through state and 2020 " - "PUMA; exact block, tract, county, and congressional district " - "cannot be recovered for a source microrecord." - ), - "treatment": ( - ( - "Retain each ACS record's observed PUMA and assign its " - "119th-CD/county from official population-weighted PUMA " - "overlaps using the recorded seed. Donor records keep " - "their certified block-ladder district/county, with PUMA " - "derived exactly from the assigned 2020 tract. Do not " - "synthesize ACS block or tract." - ) - if geography.get("donor_geography") == "preserved_assigned" - else ( - "Retain each ACS record's observed PUMA, draw ASEC PUMA " - "within native state, and assign 119th-CD/county from " - "official population-weighted PUMA overlaps using the " - "recorded seed. Do not synthesize block or tract." - ) - ), - "assignment_seed": geography.get("seed"), - "layer_vintages": geography.get("layer_vintages", {}), - "calibration_blocker": False, - }, - ] - - -def _acs_group_quarters_counts(frame: Frame) -> dict[str, int]: - household = frame.table("household") - person = frame.table("person") - household_tag = spine_column("household") - required_household = {"household_id", "TYPEHUGQ", household_tag} - required_person = { - "person_household_id", - "person_spm_unit_id", - spine_column("person"), - } - missing = sorted( - (required_household - set(household.columns)) - | (required_person - set(person.columns)) - ) - if missing: - raise SystemExit( - "Cannot summarize reviewed ACS group-quarters limitations; " - f"combined frame lacks {missing}." - ) - kind = pd.to_numeric(household["TYPEHUGQ"], errors="coerce") - household_mask = household[household_tag].eq(ACS_2024_1YR_SPINE) & kind.isin([2, 3]) - household_ids = household.loc[household_mask, "household_id"] - person_mask = person[spine_column("person")].eq(ACS_2024_1YR_SPINE) & person[ - "person_household_id" - ].isin(household_ids) - return { - "household": int(household_mask.sum()), - "person": int(person_mask.sum()), - "spm_unit": int(person.loc[person_mask, "person_spm_unit_id"].nunique()), - } - - -def _build_summary( - *, - args: argparse.Namespace, - summary_path: Path, - manifest: acs_sources.AcsSourceManifest, - source: AcsPumsSource, - base_rows: dict[str, int], - base_mass: float, - base_sha256: str, - donor_release: dict[str, object] | None, - manifest_sha256: str, - result: AcsMultispineResult, - weights_audit: dict[str, object], - transfer_coverage: dict[str, object], - input_null_audit: list[dict[str, object]], - staging_export_peak_bytes: int, - puma_ladder: UsPumaLadder, - puma_ladder_sha256: str, -) -> dict[str, object]: - output_rows = _row_counts(result.frame) - output_mass = float(result.frame.weights_for("household").total) - return { - "version": 1, - "stage": "acs_2024_1yr_multispine_base", - "artifact_kind": "nullable_precalibration_staging_h5", - "calibration_applied": False, - "simulation_ready": False, - "simulation_ready_except_calibration": True, - "simulation_readiness_blockers": ["calibration_not_applied"], - "reviewed_limitations": _reviewed_limitations( - result, - transfer_coverage=transfer_coverage, - input_null_audit=input_null_audit, - ), - "period": args.period, - "base": { - "path": str(args.base_h5.resolve()), - "sha256": base_sha256, - "rows": base_rows, - "household_weight_total": base_mass, - "donor_release": donor_release, - }, - "acs_sources": _source_provenance( - manifest, - source, - manifest_path=args.source_manifest, - manifest_sha256=manifest_sha256, - ), - "geography_ladder": { - "path": str(args.puma_ladder.resolve()), - "sha256": puma_ladder_sha256, - "pumas": len(puma_ladder), - "layer_vintages": puma_ladder.layer_vintages, - "seed": args.geography_seed, - "assignment": result.provenance["geography_ladder"], - }, - "orchestration": { - "chunksize": args.chunksize, - "acs_share": args.acs_share, - "max_households": args.max_households, - "seed": args.seed, - "geography_seed": args.geography_seed, - "n_estimators": args.n_estimators, - "max_targets_per_fit": args.max_targets_per_fit, - "donor_channel": args.donor_channel, - "provenance": result.provenance, - }, - "weights_audit": weights_audit, - "transfer_coverage": transfer_coverage, - "reviewed_engine_input_nulls": input_null_audit, - "staging_export_peak_estimate_bytes": staging_export_peak_bytes, - "rows": { - "base": base_rows, - "combined": output_rows, - }, - "household_weight_totals": { - "base": base_mass, - "combined": output_mass, - }, - "spine_totals": _spine_totals(result.frame), - "output": { - "path": str(args.out_h5.resolve()), - "sha256": _sha256(args.out_h5), - "summary_path": str(summary_path.resolve()), - "rows": output_rows, - "household_weight_total": output_mass, - }, - } - - -def _source_provenance( - manifest: acs_sources.AcsSourceManifest, - source: AcsPumsSource, - *, - manifest_path: Path | None, - manifest_sha256: str, -) -> dict[str, object]: - resolved_manifest_path = _manifest_file(manifest_path) - local_paths = { - "household": source.household_zip.resolve(), - "person": source.person_zip.resolve(), - } - return { - "manifest": ( - str(resolved_manifest_path) - if manifest_path is not None - else _PACKAGED_MANIFEST_REFERENCE - ), - "manifest_sha256": manifest_sha256, - "version": manifest.version, - "spine": manifest.spine, - "vintage": manifest.vintage, - "verified_on": manifest.verified_on, - "source_directory": manifest.source_directory, - "artifacts": [ - { - "role": artifact.role, - "filename": artifact.filename, - "url": artifact.url, - "sha256": artifact.sha256, - "size_bytes": artifact.size_bytes, - "local_path": str(local_paths[artifact.role]), - } - for artifact in manifest.artifacts - ], - } - - -def _manifest_file(override: Path | None) -> Path: - if override is not None: - return override.resolve() - return Path(acs_sources.__file__).with_name("acs_2024_1yr_sources.json") - - -def _validate_artifact_paths( - args: argparse.Namespace, - *, - summary_path: Path, -) -> None: - base = args.base_h5.resolve() - output = args.out_h5.resolve() - summary = summary_path.resolve() - if args.out_h5.suffix != ".h5": - raise SystemExit(f"--out-h5 must end with .h5, got {args.out_h5.name!r}.") - if base == output: - raise SystemExit("--out-h5 must differ from --base-h5.") - if summary in {base, output}: - raise SystemExit("--summary must differ from both --base-h5 and --out-h5.") - ladder = args.puma_ladder.resolve() - if ladder == output: - raise SystemExit("--puma-ladder must differ from --out-h5.") - - -def _spine_totals(frame: Frame) -> dict[str, dict[str, Any]]: - spine_values: set[str] = set() - for entity in frame.entities: - column = spine_column(entity) - table = frame.table(entity) - if column not in table: - raise ValueError(f"Combined ACS base lacks required spine tag {column!r}.") - if table[column].isna().any(): - raise ValueError( - f"Combined ACS base carries missing values in spine tag {column!r}." - ) - spine_values.update(map(str, table[column].dropna().unique())) - - household = frame.table("household") - household_spine = household[spine_column("household")] - weights = pd.Series( - frame.weights_for("household").values, - index=household.index, - ) - return { - spine: { - "rows": { - entity: int(frame.table(entity)[spine_column(entity)].eq(spine).sum()) - for entity in frame.entities - }, - "household_weight_total": float( - weights.loc[household_spine.eq(spine)].sum() - ), - } - for spine in sorted(spine_values) - } +Use the sha-pinned pool builder instead: + uv run tools/build_us_multispine_pool.py --help +""" def _load_base_frame(path: Path) -> Frame: - """Load the dense donor H5 without importing PolicyEngine-US at tool import.""" + """Load a legacy calibrated US H5 for the deprecated local-release lane.""" - from policyengine_us.data import USSingleYearDataset - - dataset = USSingleYearDataset(file_path=str(path)) - tables = { - "person": dataset.person, - "household": dataset.household, - "tax_unit": dataset.tax_unit, - "spm_unit": dataset.spm_unit, - "family": dataset.family, - "marital_unit": dataset.marital_unit, - } - household_weights = ( - tables["household"].pop("household_weight").to_numpy(dtype=np.float64) - ) - return Frame( - tables, - US_SCHEMA, - { - "household": Weights( - household_weights, - WeightKind.CALIBRATED, - ) - }, - ) + return load_legacy_calibrated_us_h5(path) def _write_dataset( @@ -995,177 +42,24 @@ def _write_dataset( path: Path, *, period: int, - artifact_kind: str = "nullable_precalibration_staging_h5", + artifact_kind: str = LEGACY_NULLABLE_STAGING_ARTIFACT_KIND, ) -> None: - """Write a populace US H5 with one-table-at-a-time verification.""" - - output = Path(path) - output.unlink(missing_ok=True) - try: - with pd.HDFStore(output, mode="w") as store: - for entity in frame.entities: - table = frame.table(entity) - if entity == "household": - table = table.copy() - table["household_weight"] = frame.weights_for("household").values - if len(table): - # Fixed format preserves mixed bool/null object columns - # losslessly. Table format rejects them, which would force - # an unauthorized fill or type rewrite on base-only inputs. - with warnings.catch_warnings(): - warnings.simplefilter("ignore", pd.errors.PerformanceWarning) - store.put(entity, table, format="fixed") - store.put( - "_time_period", - pd.Series([int(period)]), - format="table", - ) - store.put( - "_populace_staging_metadata", - pd.Series( - [ - json.dumps( - { - "artifact_kind": artifact_kind, - "entity_hdf_format": "fixed_nullable", - "household_weight_kind": frame.weights_for( - "household" - ).kind.value, - }, - sort_keys=True, - ) - ] - ), - format="table", - ) - - with pd.HDFStore(output, mode="r") as store: - for entity in frame.entities: - expected = frame.table(entity) - if not len(expected): - continue - stored = store[entity] - expected_columns = list(expected.columns) - if entity == "household": - expected_columns.append("household_weight") - if ( - len(stored) != len(expected) - or list(stored.columns) != expected_columns - ): - raise RuntimeError( - f"Staging H5 round trip changed {entity!r}: expected " - f"{len(expected)} rows/{expected_columns}, got " - f"{len(stored)} rows/{list(stored.columns)}." - ) - del stored - except BaseException: - output.unlink(missing_ok=True) - raise - - -def _engine_input_null_audit( - frame: Frame, - engine: Any | None = None, -) -> list[dict[str, object]]: - """Inventory nullable engine inputs for the reviewed-limitations summary.""" + """Write the legacy lane's nullable H5 through the shared atomic writer.""" - if engine is None: - from populace.frame.adapters.policyengine_us import PolicyEngineUSEngine - - engine = PolicyEngineUSEngine() - input_names = set(engine.variables()) - entries: list[dict[str, object]] = [] - for entity in frame.entities: - table = frame.table(entity) - tag = spine_column(entity) - for column in sorted(set(table.columns).intersection(input_names)): - missing = table[column].isna() - if not missing.any(): - continue - by_spine: dict[str, int] = {} - if tag in table: - by_spine = { - str(spine): int((missing & table[tag].eq(spine)).sum()) - for spine in sorted(map(str, table[tag].dropna().unique())) - if int((missing & table[tag].eq(spine)).sum()) - } - entries.append( - { - "entity": entity, - "column": column, - "dtype": engine.variable_metadata(column).dtype, - "missing_rows": int(missing.sum()), - "rows": len(table), - "missing_rows_by_spine": by_spine, - } - ) - return entries - - -def _preflight_staging_export( - frame: Frame, - *, - max_peak_bytes: int = DEFAULT_STAGING_EXPORT_PEAK_LIMIT_BYTES, -) -> int: - if type(max_peak_bytes) is not int or max_peak_bytes <= 0: - raise ValueError("max_peak_bytes must be a positive integer.") - table_bytes = { - entity: int(frame.table(entity).memory_usage(index=True, deep=True).sum()) - for entity in frame.entities - } - resident = sum(table_bytes.values()) - resident += sum( - frame.weights_for(entity).values.nbytes for entity in frame.weighted_entities - ) - resident += int(frame.strata.memory_usage(index=True, deep=True)) - largest_table = max(table_bytes.values(), default=0) - household_copy = table_bytes.get("household", 0) + 8 * frame.n("household") - estimate = int( - resident - + 2 * largest_table - + household_copy - + _STAGING_EXPORT_FIXED_OVERHEAD_BYTES + write_nullable_us_h5( + frame, + path, + period=period, + artifact_kind=artifact_kind, ) - if estimate > max_peak_bytes: - raise MemoryError( - "Nullable ACS staging export is estimated to require " - f"{estimate / 1_000_000_000:.2f} GB, above the " - f"{max_peak_bytes / 1_000_000_000:.2f} GB limit." - ) - return estimate - -def _row_counts(frame: Frame) -> dict[str, int]: - return {entity: frame.n(entity) for entity in frame.entities} - - -def _sha256(path: Path) -> str: - digest = hashlib.sha256() - with path.open("rb") as file: - while chunk := file.read(1024 * 1024): - digest.update(chunk) - return digest.hexdigest() - - -def _positive_int(value: str) -> int: - parsed = int(value) - if parsed <= 0: - raise argparse.ArgumentTypeError("must be a positive integer") - return parsed - - -def _nonnegative_int(value: str) -> int: - parsed = int(value) - if parsed < 0: - raise argparse.ArgumentTypeError("must be a non-negative integer") - return parsed +def main(argv: list[str] | None = None) -> int: + """Refuse the retired CLI and name the explicit migration command.""" -def _open_unit_interval(value: str) -> float: - parsed = float(value) - if not 0.0 < parsed < 1.0: - raise argparse.ArgumentTypeError("must be strictly between 0 and 1") - return parsed + del argv + print(_MIGRATION_MESSAGE, file=sys.stderr) + return 2 if __name__ == "__main__": From 36fc80012a8208713a1730ae1e05b22094533d75 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 20:34:50 -0400 Subject: [PATCH 07/94] Preserve raw values in ACS transfer --- .../populace/build/us_runtime/acs_transfer.py | 332 ++++++++++++++---- .../tests/test_us_acs_transfer.py | 299 +++++++++++++++- 2 files changed, 550 insertions(+), 81 deletions(-) diff --git a/packages/populace-build/src/populace/build/us_runtime/acs_transfer.py b/packages/populace-build/src/populace/build/us_runtime/acs_transfer.py index 1d58569b..bb12656f 100644 --- a/packages/populace-build/src/populace/build/us_runtime/acs_transfer.py +++ b/packages/populace-build/src/populace/build/us_runtime/acs_transfer.py @@ -2,9 +2,9 @@ ACS observes a dense demographic and geography spine but not every input the US model consumes. This module learns missing numeric and boolean leaves from -an ASEC x PUF donor with :mod:`populace.fit`. Native ACS columns always win: a -requested target already present on the recipient is never overwritten, and -model-only predictors are never added to the returned frame. +an ASEC x PUF donor with :mod:`populace.fit`. Native values always win: only +null target cells are filled, and model-only predictors are never added to the +returned frame. Optional ACS predictors keep their source missingness. Recipient rows are partitioned by the predictors they actually observe, then one weighted QRF is @@ -12,8 +12,8 @@ No missing value is ever converted to zero merely to satisfy an estimator. Imputation provenance is external to the frame. ``imputed_inputs`` records one -immutable entry per added column, including every availability-pattern fit; it -is not an export column and cannot leak into a PolicyEngine dataset. +immutable entry per filled column, including every availability-pattern fit; +it is not an export column and cannot leak into a PolicyEngine dataset. """ from __future__ import annotations @@ -338,6 +338,8 @@ class AcsImputedInput: seed: int weight_kind: str patterns: tuple[AcsTransferPattern, ...] = () + #: Number of recipient null cells this operation filled. + imputed_recipient_rows: int = 0 unmodeled_recipient_rows: int = 0 #: Non-fit derivation kind for deterministic post-transfer columns #: (e.g. ``"split_component_by_share"``); ``None`` for QRF-fitted ones. @@ -374,7 +376,6 @@ class _FamilyFit: predictors: tuple[str, ...] weight_kind: str family_seed: int - unmodeled_recipient_rows: int target_encodings: Mapping[str, _TargetEncoding] @@ -481,6 +482,8 @@ def derive_acs_schedule_d_capital_gain_distributions( def reconcile_acs_adult_care( person: pd.DataFrame, + *, + mutable_rows: np.ndarray | pd.Series | None = None, ) -> tuple[pd.Series, dict[str, int]]: """Reconcile transferred adult-care expenses to the statute structure. @@ -489,10 +492,25 @@ def reconcile_acs_adult_care( carrier; independent person-grain predictions cannot. Deterministically: clear expenses on non-qualifying people, then keep only the largest carrier per unit (ties broken by row order). + + When ``mutable_rows`` is supplied, only those newly imputed expense cells + may change. A pre-existing positive carrier blocks every mutable carrier + in its unit, preserving raw values while preventing the transfer from + introducing an additional carrier. """ flag = person[_ADULT_CARE_FLAG].fillna(False).astype(bool) - expenses = pd.to_numeric(person[_ADULT_CARE_EXPENSE], errors="coerce").fillna(0.0) + raw_expenses = pd.to_numeric(person[_ADULT_CARE_EXPENSE], errors="coerce") + expenses = raw_expenses.fillna(0.0) + if mutable_rows is None: + mutable = pd.Series(True, index=person.index) + else: + mutable_array = np.asarray(mutable_rows, dtype=bool) + if mutable_array.ndim != 1 or len(mutable_array) != len(person): + raise ValueError( + "mutable_rows must be a one-dimensional mask aligned to person." + ) + mutable = pd.Series(mutable_array, index=person.index) role = person[_ADULT_CARE_ROLE].astype(str) units = person[_ADULT_CARE_UNIT] is_dependent = role.eq("DEPENDENT") @@ -501,22 +519,27 @@ def reconcile_acs_adult_care( unit_married = is_spouse.groupby(units).transform("any") qualifying = flag & (is_dependent | ((is_head | is_spouse) & unit_married)) - positive = expenses > 0.0 - cleared_ineligible = int((positive & ~qualifying).sum()) - expenses = expenses.where(qualifying, 0.0) + mutable_positive = mutable & (expenses > 0.0) + cleared_ineligible_mask = mutable_positive & ~qualifying + cleared_ineligible = int(cleared_ineligible_mask.sum()) - positive = expenses > 0.0 - rank = (-expenses).groupby(units).rank(method="first") - keep = positive & rank.eq(1.0) - cleared_multi_carrier = int((positive & ~keep).sum()) - expenses = expenses.where(keep, 0.0) + candidates = mutable_positive & qualifying + immutable_positive = ~mutable & (expenses > 0.0) + unit_has_immutable = immutable_positive.groupby(units).transform("any") + ranked = (-expenses.where(candidates, 0.0)).groupby(units).rank(method="first") + keep = candidates & ~unit_has_immutable & ranked.eq(1.0) + cleared_multi_carrier = int((candidates & ~keep).sum()) + + reconciled = expenses.copy() if mutable_rows is None else raw_expenses.copy() + reconciled.loc[keep] = expenses.loc[keep] + reconciled.loc[mutable & ~keep] = 0.0 counts = { "cleared_ineligible_carriers": cleared_ineligible, "cleared_multi_carrier_rows": cleared_multi_carrier, - "remaining_carriers": int((expenses > 0.0).sum()), + "remaining_carriers": int((reconciled > 0.0).sum()), } - return expenses, counts + return reconciled, counts def declared_acs_transfer_target_families() -> TargetFamilies: @@ -655,8 +678,9 @@ def transfer_acs_inputs( ``target_families`` is an ``entity -> family -> targets`` mapping. Each family is split into recipient optional-predictor availability patterns. - Every pattern gets a separately seeded QRF fit on donor rows finite for - exactly the predictors that pattern observes. + Every pattern gets a separately seeded QRF fit on donor rows complete for + every family target and finite for exactly the predictors that pattern + observes. Existing non-null target cells remain unchanged. Families wider than ``max_targets_per_fit`` are deterministically split into bounded chained-QRF batches so fitted forests cannot grow linearly @@ -723,24 +747,45 @@ def transfer_acs_inputs( } provenance: list[AcsImputedInput] = [] fit_records: list[FitWeightRecord] = [] + imputed_masks: dict[tuple[str, str], np.ndarray] = {} for entity, family, targets in active: + recipient_table = recipient.table(entity) + target_missing = { + target: ( + recipient_table[target].isna().to_numpy(dtype=bool) + if target in recipient_table.columns + else np.ones(len(recipient_table), dtype=bool) + ) + for target in targets + } fitted = _fit_family_patterns( fit_donor, recipient, entity=entity, family=family, targets=targets, + target_missing=target_missing, seed=seed, n_estimators=n_estimators, ) for target in targets: - output_tables[entity][target] = _prediction_values( + predicted = _prediction_values( fitted.predictions[target], encoding=fitted.target_encodings[target], entity=entity, target=target, ) + merged, imputed = _fill_recipient_nulls( + output_tables[entity], + target=target, + predicted=predicted, + ) + output_tables[entity][target] = merged + imputed_masks[(entity, target)] = imputed + missing_rows = target_missing[target] + if not missing_rows.any(): + continue provenance.append( AcsImputedInput( column=target, @@ -752,7 +797,8 @@ def transfer_acs_inputs( seed=fitted.family_seed, weight_kind=fitted.weight_kind, patterns=fitted.patterns, - unmodeled_recipient_rows=fitted.unmodeled_recipient_rows, + imputed_recipient_rows=int(imputed.sum()), + unmodeled_recipient_rows=int((missing_rows & ~imputed).sum()), ) ) fit_records.extend(fitted.fit_records) @@ -760,6 +806,7 @@ def transfer_acs_inputs( _apply_post_transfer_structure( output_tables, provenance, + imputed_masks=imputed_masks, donor_spine=donor_spine, resolved_channel=resolved_channel, ) @@ -790,13 +837,14 @@ def _apply_post_transfer_structure( output_tables: dict[str, pd.DataFrame], provenance: list[AcsImputedInput], *, + imputed_masks: Mapping[tuple[str, str], np.ndarray], donor_spine: str, resolved_channel: str | None, ) -> None: """Apply the deterministic post-fit steps the base's construction implies. - Both steps key off what THIS transfer produced (the provenance list), so - custom test plans that never touch these families are unaffected: + Both steps key off cells THIS transfer filled, so custom test plans that + never touch these families are unaffected: - The Schedule D CGD memo leg is derived from the two transferred capital-gain parents at the packaged share with route exclusivity. @@ -807,41 +855,79 @@ def _apply_post_transfer_structure( person = output_tables.get("person") if person is None: return - transferred = {item.column for item in provenance if item.entity == "person"} cgd_parents = {_SCHEDULE_D_CGD_SOURCE, _SCHEDULE_D_CGD_EXCLUSIVE_WITH} - if cgd_parents <= transferred: + cgd_candidate = np.zeros(len(person), dtype=bool) + for parent in cgd_parents: + cgd_candidate |= imputed_masks.get( + ("person", parent), + np.zeros(len(person), dtype=bool), + ) + if cgd_candidate.any() and cgd_parents <= set(person.columns): + source = pd.to_numeric(person[_SCHEDULE_D_CGD_SOURCE], errors="coerce") + other_route = pd.to_numeric( + person[_SCHEDULE_D_CGD_EXCLUSIVE_WITH], + errors="coerce", + ) + derivable = ( + cgd_candidate + & np.isfinite(source.to_numpy(dtype=np.float64)) + & np.isfinite(other_route.to_numpy(dtype=np.float64)) + ) if _SCHEDULE_D_CGD_COLUMN in person.columns: - raise ValueError( - f"{_SCHEDULE_D_CGD_COLUMN!r} must not be fitted or natively " - "present; it is derived from its transferred parents." + derived_output = person[_SCHEDULE_D_CGD_COLUMN].copy() + else: + derived_output = pd.Series( + np.nan, + index=person.index, + name=_SCHEDULE_D_CGD_COLUMN, ) - values, derivation = derive_acs_schedule_d_capital_gain_distributions(person) - person[_SCHEDULE_D_CGD_COLUMN] = values - provenance.append( - AcsImputedInput( - column=_SCHEDULE_D_CGD_COLUMN, - entity="person", - family="capital_gain_details", - donor_spine=donor_spine, - donor_channel=resolved_channel, - predictors=( - _SCHEDULE_D_CGD_SOURCE, - _SCHEDULE_D_CGD_EXCLUSIVE_WITH, - ), - seed=0, - weight_kind="deterministic", - derivation="split_component_by_share", - reconciliation={ - key: int(value) - for key, value in derivation.items() - if isinstance(value, int) - }, + fill = derivable & derived_output.isna().to_numpy(dtype=bool) + if not fill.any(): + derivation: dict[str, object] = {} + else: + values, derivation = derive_acs_schedule_d_capital_gain_distributions( + person.loc[fill] + ) + derived_output.loc[fill] = values + person[_SCHEDULE_D_CGD_COLUMN] = derived_output + if fill.any(): + provenance.append( + AcsImputedInput( + column=_SCHEDULE_D_CGD_COLUMN, + entity="person", + family="capital_gain_details", + donor_spine=donor_spine, + donor_channel=resolved_channel, + predictors=( + _SCHEDULE_D_CGD_SOURCE, + _SCHEDULE_D_CGD_EXCLUSIVE_WITH, + ), + seed=0, + weight_kind="deterministic", + imputed_recipient_rows=int(fill.sum()), + unmodeled_recipient_rows=int( + ( + cgd_candidate + & derived_output.isna().to_numpy(dtype=bool) + & ~derivable + ).sum() + ), + derivation="split_component_by_share", + reconciliation={ + key: int(value) + for key, value in derivation.items() + if isinstance(value, int) + }, + ) ) - ) - adult_care = {_ADULT_CARE_FLAG, _ADULT_CARE_EXPENSE} - if adult_care <= transferred: + adult_care_expense_mask = imputed_masks.get(("person", _ADULT_CARE_EXPENSE)) + if ( + adult_care_expense_mask is not None + and adult_care_expense_mask.any() + and _ADULT_CARE_FLAG in person.columns + ): structural = {_ADULT_CARE_ROLE, _ADULT_CARE_UNIT} missing = sorted(structural - set(person.columns)) if missing: @@ -850,7 +936,10 @@ def _apply_post_transfer_structure( f"column(s) {missing}; refusing to ship an unreconciled " "expense surface." ) - expenses, counts = reconcile_acs_adult_care(person) + expenses, counts = reconcile_acs_adult_care( + person, + mutable_rows=adult_care_expense_mask, + ) person[_ADULT_CARE_EXPENSE] = expenses for index, item in enumerate(provenance): if item.column == _ADULT_CARE_EXPENSE and item.entity == "person": @@ -865,13 +954,22 @@ def _fit_family_patterns( entity: str, family: str, targets: tuple[str, ...], + target_missing: Mapping[str, np.ndarray], seed: int, n_estimators: int, ) -> _FamilyFit: _validate_donor_targets(donor, entity=entity, targets=targets) - target_encodings = _target_encodings( - donor.table(entity), + donor_table = donor.table(entity) + target_complete = _complete_target_mask(donor_table, targets=targets) + if not target_complete.any(): + raise ValueError( + f"ACS transfer family {family!r} on entity {entity!r} has no " + f"donor rows complete for every target {list(targets)}." + ) + target_encodings = _complete_case_target_encodings( + donor_table, targets=targets, + complete=target_complete, ) model_targets = tuple( dict.fromkeys(encoding.model_target for encoding in target_encodings.values()) @@ -890,7 +988,13 @@ def _fit_family_patterns( ) recipient_required = surface.recipient.loc[:, list(surface.required)] - eligible = np.isfinite(recipient_required.to_numpy(dtype=np.float64)).all(axis=1) + needs_prediction = np.logical_or.reduce( + [np.asarray(target_missing[target], dtype=bool) for target in targets] + ) + eligible = ( + np.isfinite(recipient_required.to_numpy(dtype=np.float64)).all(axis=1) + & needs_prediction + ) if not eligible.any(): raise ValueError( f"ACS transfer family {family!r} on entity {entity!r} has no " @@ -910,7 +1014,7 @@ def _fit_family_patterns( ): predictors = (*surface.required, *observed_optional) donor_matrix = surface.donor.loc[:, list(predictors)].to_numpy(dtype=np.float64) - donor_mask = np.isfinite(donor_matrix).all(axis=1) + donor_mask = np.isfinite(donor_matrix).all(axis=1) & target_complete donor_rows = int(donor_mask.sum()) if donor_rows == 0: raise ValueError( @@ -1000,7 +1104,6 @@ def _fit_family_patterns( predictors=used_predictors, weight_kind=next(iter(kinds)), family_seed=family_seed, - unmodeled_recipient_rows=int((~eligible).sum()), target_encodings=target_encodings, ) @@ -1610,9 +1713,12 @@ def _missing_target_families( ) -> list[tuple[str, str, tuple[str, ...]]]: active: list[tuple[str, str, tuple[str, ...]]] = [] for entity, family, targets in families: - missing: list[str] = [] + incomplete: list[str] = [] + table = recipient.table(entity) for target in targets: - if target in recipient.table(entity).columns: + if target in table.columns: + if table[target].isna().any(): + incomplete.append(target) continue owner = _column_owner_or_none(recipient, target) if owner is not None: @@ -1621,9 +1727,22 @@ def _missing_target_families( f"{entity!r}, but the recipient already carries it on " f"entity {owner!r}." ) - missing.append(target) - if missing: - active.append((entity, family, tuple(missing))) + incomplete.append(target) + + # A joint codec needs both columns to learn only observed donor pairs. + # The complete companion is fitted but its recipient values are not + # written because target-level null masks remain authoritative. + immigration_pair = set(_IMMIGRATION_STATUS_TARGETS) + if immigration_pair.issubset(targets) and immigration_pair.intersection( + incomplete + ): + incomplete = [ + target + for target in targets + if target in immigration_pair or target in incomplete + ] + if incomplete: + active.append((entity, family, tuple(incomplete))) return active @@ -1784,21 +1903,6 @@ def _validate_donor_targets( f"ACS transfer donor targets on entity {entity!r} must be numeric, " f"boolean, or categorical string/enum values: {unsupported}." ) - non_finite = [] - for target in targets: - values = table[target] - if values.isna().any(): - non_finite.append(target) - elif ( - _is_numeric_or_bool(values) - and not np.isfinite(_as_float_array(values)).all() - ): - non_finite.append(target) - if non_finite: - raise ValueError( - f"ACS transfer donor targets on entity {entity!r} contain " - f"non-finite values: {non_finite}." - ) wrong_engine_entity = { target: metadata.entity for target in targets @@ -1812,6 +1916,23 @@ def _validate_donor_targets( ) +def _complete_target_mask( + table: pd.DataFrame, + *, + targets: Sequence[str], +) -> np.ndarray: + """Rows observed and finite for every target in one chained fit.""" + + complete = np.ones(len(table), dtype=bool) + for target in targets: + values = table[target] + observed = values.notna().to_numpy(dtype=bool, copy=True) + if _is_numeric_or_bool(values): + observed &= np.isfinite(_as_float_array(values)) + complete &= observed + return complete + + def _is_supported_target(series: pd.Series) -> bool: if _is_numeric_or_bool(series): return True @@ -1901,6 +2022,29 @@ def _target_encodings( return result +def _complete_case_target_encodings( + table: pd.DataFrame, + *, + targets: Sequence[str], + complete: np.ndarray, +) -> dict[str, _TargetEncoding]: + """Learn target support on complete cases, then restore donor alignment.""" + + complete_table = table.iloc[np.flatnonzero(complete)] + encodings = _target_encodings(complete_table, targets=targets) + result: dict[str, _TargetEncoding] = {} + positions = np.flatnonzero(complete) + for target, encoding in encodings.items(): + aligned = pd.Series( + np.nan, + index=table.index, + name=encoding.model_values.name, + ) + aligned.iloc[positions] = encoding.model_values.to_numpy(dtype=np.float64) + result[target] = replace(encoding, model_values=aligned) + return result + + def _target_encoding(series: pd.Series, *, target: str) -> _TargetEncoding: """Encode one scalar donor target for QRF without changing its support.""" @@ -2099,6 +2243,38 @@ def _prediction_values( return result +def _fill_recipient_nulls( + table: pd.DataFrame, + *, + target: str, + predicted: np.ndarray | pd.api.extensions.ExtensionArray, +) -> tuple[pd.Series, np.ndarray]: + """Merge predictions into null cells without changing observed values.""" + + prediction = pd.Series(predicted, index=table.index, name=target) + predicted_observed = prediction.notna().to_numpy(dtype=bool) + if target not in table.columns: + return prediction, predicted_observed + + result = table[target].copy() + imputed = result.isna().to_numpy(dtype=bool) & predicted_observed + if not imputed.any(): + return result, imputed + + if isinstance(result.dtype, pd.CategoricalDtype): + additions = [ + value + for value in pd.unique(prediction.iloc[np.flatnonzero(imputed)]) + if value not in result.cat.categories + ] + if additions: + result = result.cat.add_categories(additions) + + positions = np.flatnonzero(imputed) + result.iloc[positions] = prediction.iloc[positions].to_numpy() + return result, imputed + + def _snap_to_support(values: np.ndarray, support: np.ndarray) -> np.ndarray: """Map values to nearest sorted donor support in O(n log k) memory.""" diff --git a/packages/populace-build/tests/test_us_acs_transfer.py b/packages/populace-build/tests/test_us_acs_transfer.py index 5086a7e9..23a2a888 100644 --- a/packages/populace-build/tests/test_us_acs_transfer.py +++ b/packages/populace-build/tests/test_us_acs_transfer.py @@ -252,6 +252,19 @@ def _with_columns( ) +def _with_metadata(frame: Frame, metadata: dict[str, object]) -> Frame: + tables = {name: frame.table(name).copy() for name in frame.entities} + tables.update({name: frame.link(name).copy() for name in frame.links}) + return Frame( + tables, + frame.schema, + {name: frame.weights_for(name) for name in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=metadata, + ) + + def _with_full_us_schema(frame: Frame) -> Frame: """Promote the compact transfer fixture to all PolicyEngine-US grains.""" @@ -1112,6 +1125,133 @@ def test_explicit_family_is_seed_deterministic_and_preserves_recipient_index() - assert first.imputed_inputs[0].donor_channel == "puf_tax_detail" +def test_nullable_recipient_target_fills_only_nulls_and_preserves_metadata( + monkeypatch: pytest.MonkeyPatch, +) -> None: + recipient = _with_columns( + _recipient_frame(), + "person", + { + "qualified_dividend_income": pd.array( + [111.0, pd.NA, 333.0, pd.NA, 555.0, 666.0], + dtype="Float64", + ), + "takes_up_medicaid_if_eligible": pd.array( + [True, pd.NA, False, pd.NA, True, False], + dtype="boolean", + ), + }, + ) + recipient = _with_metadata( + recipient, + { + "assembly_receipt": { + "channels": ("asec", "acs"), + "id_bound": 10_000, + } + }, + ) + before = { + target: recipient.person[target].copy() + for target in ( + "qualified_dividend_income", + "takes_up_medicaid_if_eligible", + ) + } + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + recipient, + _donor_frame(), + target_families={ + "person": { + "tax_detail": ( + "qualified_dividend_income", + "takes_up_medicaid_if_eligible", + ), + }, + }, + n_estimators=1, + ) + + for target, target_before in before.items(): + after = result.frame.person[target] + observed = target_before.notna() + pd.testing.assert_series_equal( + after.loc[observed], + target_before.loc[observed], + ) + assert after.loc[~observed].notna().all() + assert recipient.person[target].isna().sum() == 2 + assert result.frame.metadata == recipient.metadata + provenance = {item.column: item for item in result.imputed_inputs} + assert set(provenance) == set(before) + assert all(item.imputed_recipient_rows == 2 for item in provenance.values()) + assert all(item.unmodeled_recipient_rows == 0 for item in provenance.values()) + assert all( + sum(pattern.recipient_rows for pattern in item.patterns) == 2 + for item in provenance.values() + ) + + +def test_donor_family_fit_uses_rows_complete_for_every_target( + monkeypatch: pytest.MonkeyPatch, +) -> None: + donor = _with_columns( + _donor_frame(), + "person", + { + "nullable_target_a": [ + np.nan, + 2.0, + 3.0, + 4.0, + 5.0, + 6.0, + 7.0, + 8.0, + ], + "nullable_target_b": [ + 10.0, + np.nan, + 30.0, + 40.0, + 50.0, + 60.0, + 70.0, + 80.0, + ], + }, + ) + recipient = _recipient_frame() + for column in ( + "employment_income_before_lsr", + "self_employment_income_before_lsr", + ): + donor = _drop_column(donor, "person", column) + recipient = _drop_column(recipient, "person", column) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + recipient, + donor, + target_families={ + "person": { + "nullable_pair": ("nullable_target_a", "nullable_target_b"), + }, + }, + n_estimators=1, + ) + + assert len(_MeanQRF.calls) == 1 + fitted_targets = _MeanQRF.calls[0]["targets"] + assert len(fitted_targets) == 6 + assert np.isfinite(fitted_targets.to_numpy(dtype=np.float64)).all() + assert {item.patterns[0].donor_rows for item in result.imputed_inputs} == {6} + + def test_formula_owned_target_is_refused_before_fit() -> None: with pytest.raises(ValueError, match="formula-owned.*interest_deduction"): transfer_acs_inputs( @@ -1124,17 +1264,17 @@ def test_formula_owned_target_is_refused_before_fit() -> None: ) -def test_non_finite_donor_target_is_refused_without_zero_fill() -> None: +def test_all_missing_donor_target_is_refused_without_zero_fill() -> None: donor = _replace_column( _donor_frame(), "person", "qualified_dividend_income", - [900.0, 200.0, np.nan, 20.0, 3_000.0, 1_500.0, 8_000.0, 40.0], + [np.nan] * 8, ) with pytest.raises( ValueError, - match="donor targets.*non-finite.*qualified_dividend_income", + match="no donor rows complete for every target.*qualified_dividend_income", ): transfer_acs_inputs( _recipient_frame(), @@ -1146,6 +1286,159 @@ def test_non_finite_donor_target_is_refused_without_zero_fill() -> None: ) +def test_schedule_d_post_transfer_fills_only_newly_imputed_rows( + monkeypatch: pytest.MonkeyPatch, +) -> None: + donor = _with_columns( + _donor_frame(), + "person", + { + "long_term_capital_gains_before_response": [ + 100.0, + 200.0, + 300.0, + 400.0, + 500.0, + 600.0, + 700.0, + 800.0, + ], + "non_sch_d_capital_gains": [0.0] * 8, + }, + ) + recipient = _with_columns( + _recipient_frame(), + "person", + { + "long_term_capital_gains_before_response": [ + 1_000.0, + np.nan, + 2_000.0, + np.nan, + 3_000.0, + np.nan, + ], + "non_sch_d_capital_gains": [ + 0.0, + np.nan, + 500.0, + np.nan, + 0.0, + np.nan, + ], + "schedule_d_capital_gain_distributions": [ + 777.0, + np.nan, + 222.0, + 888.0, + 333.0, + np.nan, + ], + }, + ) + cgd_before = recipient.person["schedule_d_capital_gain_distributions"].copy() + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + recipient, + donor, + target_families={ + "person": { + "capital_gain_details": ( + "long_term_capital_gains_before_response", + "non_sch_d_capital_gains", + ), + }, + }, + n_estimators=1, + ) + + cgd = result.frame.person["schedule_d_capital_gain_distributions"] + measured = cgd_before.notna() + pd.testing.assert_series_equal(cgd.loc[measured], cgd_before.loc[measured]) + assert cgd.iloc[[1, 5]].gt(0.0).all() + derived = next( + item + for item in result.imputed_inputs + if item.column == "schedule_d_capital_gain_distributions" + ) + assert derived.imputed_recipient_rows == 2 + + +def test_adult_care_reconciliation_changes_only_imputed_expenses( + monkeypatch: pytest.MonkeyPatch, +) -> None: + donor = _with_columns( + _donor_frame(), + "person", + { + "is_incapable_of_self_care": [True] * 8, + "pre_subsidy_care_expenses": [ + 100.0, + 200.0, + 300.0, + 400.0, + 500.0, + 600.0, + 700.0, + 800.0, + ], + }, + ) + recipient = _with_columns( + _recipient_frame(), + "person", + { + "is_incapable_of_self_care": [True] * 6, + "pre_subsidy_care_expenses": [ + 900.0, + np.nan, + 0.0, + np.nan, + 0.0, + np.nan, + ], + "tax_unit_role_input": ["DEPENDENT"] * 6, + }, + ) + before = recipient.person["pre_subsidy_care_expenses"].copy() + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + recipient, + donor, + target_families={ + "person": { + "adult_care": ( + "is_incapable_of_self_care", + "pre_subsidy_care_expenses", + ), + }, + }, + n_estimators=1, + ) + + expenses = result.frame.person["pre_subsidy_care_expenses"] + measured = before.notna() + pd.testing.assert_series_equal(expenses.loc[measured], before.loc[measured]) + assert expenses.iloc[1] == 0.0 + assert expenses.iloc[3] > 0.0 + assert expenses.iloc[5] == 0.0 + provenance = next( + item + for item in result.imputed_inputs + if item.column == "pre_subsidy_care_expenses" + ) + assert provenance.imputed_recipient_rows == 3 + assert provenance.reconciliation == { + "cleared_ineligible_carriers": 0, + "cleared_multi_carrier_rows": 2, + "remaining_carriers": 2, + } + + def test_non_finite_recipient_predictor_is_refused_without_zero_fill() -> None: recipient = _replace_column( _recipient_frame(), From a083daee40633645bda0b4e8cb87eebe1edf4daf Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 20:37:58 -0400 Subject: [PATCH 08/94] Add bounded ASEC checkpoint loader --- .../src/populace/build/us_runtime/__init__.py | 4 + .../build/us_runtime/asec_checkpoint.py | 150 ++++++++++++++ .../tests/test_us_asec_checkpoint.py | 194 ++++++++++++++++++ .../tests/test_us_spine_blindness.py | 1 + 4 files changed, 349 insertions(+) create mode 100644 packages/populace-build/src/populace/build/us_runtime/asec_checkpoint.py create mode 100644 packages/populace-build/tests/test_us_asec_checkpoint.py diff --git a/packages/populace-build/src/populace/build/us_runtime/__init__.py b/packages/populace-build/src/populace/build/us_runtime/__init__.py index 1fec8d6a..daecadd6 100644 --- a/packages/populace-build/src/populace/build/us_runtime/__init__.py +++ b/packages/populace-build/src/populace/build/us_runtime/__init__.py @@ -59,6 +59,9 @@ us_alimony_stage_spec, us_alimony_summary, ) +from populace.build.us_runtime.asec_checkpoint import ( + load_asec_pre_clone_checkpoint, +) from populace.build.us_runtime.asec_pool import ( AsecSource, build_pooled_asec_unit_frame, @@ -1796,6 +1799,7 @@ "load_default_reform_specs", "load_congressional_district_vintage_crosswalk", "load_default_congressional_district_vintage_crosswalk", + "load_asec_pre_clone_checkpoint", "load_puf_tax_unit_donor", "normalize_district_code", "parse_baf_cd_layer", diff --git a/packages/populace-build/src/populace/build/us_runtime/asec_checkpoint.py b/packages/populace-build/src/populace/build/us_runtime/asec_checkpoint.py new file mode 100644 index 00000000..991cdcfe --- /dev/null +++ b/packages/populace-build/src/populace/build/us_runtime/asec_checkpoint.py @@ -0,0 +1,150 @@ +"""Bounded loading for the input-complete ASEC pre-clone checkpoint. + +The checkpoint is produced by the outer-stage runtime after ASEC-only input +enrichment. This module validates that artifact binding and the loaded +``Frame`` boundary; it does not compute a whole-file digest. Callers that +pin input files own that separate provenance check. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +import numpy as np + +from populace.build.frame_checkpoint import load_frame_checkpoint +from populace.build.outer_stage_runtime import ( + OUTER_STAGE_CONTEXT_SCHEMA_VERSION, + FrameIdentity, + frame_identity, +) +from populace.frame import US_SCHEMA, Frame, WeightKind, Weights + +__all__ = ["load_asec_pre_clone_checkpoint"] + +_OUTER_STAGE_ARTIFACT_KIND = "populace_outer_stage_frame" +_PRE_CLONE_STAGE = "pre_clone_enrichment" +_PRE_CLONE_STAGE_INDEX = 1 +_LOWERCASE_SHA256 = re.compile(r"[0-9a-f]{64}") +_BINDING_KEYS = frozenset( + { + "artifact_kind", + "identity", + "pipeline_sha256", + "schema_version", + "stage", + "stage_index", + } +) + + +def load_asec_pre_clone_checkpoint( + path: str | Path, +) -> tuple[Frame, dict[str, object]]: + """Load and validate one input-complete ASEC pre-clone checkpoint. + + Args: + path: Explicit outer-stage Frame checkpoint produced for + ``pre_clone_enrichment``. + + Returns: + The restored US ``Frame`` and its canonical, JSON-ready outer-stage + checkpoint binding. The metadata contains no newly computed file + digest. + + Raises: + FileNotFoundError: If ``path`` does not name a checkpoint file. + ValueError: If the artifact is not bound to the required outer stage, + its stored identity differs from the loaded ``Frame``, or the + frame lacks the required US schema and positive household-only + typed weights. + """ + + checkpoint_path = Path(path) + loaded = load_frame_checkpoint(checkpoint_path) + metadata = _validate_outer_stage_binding( + loaded.metadata, + path=checkpoint_path, + ) + stored_identity = FrameIdentity.from_payload( + metadata["identity"], + label="ASEC pre-clone checkpoint identity", + ) + actual_identity = frame_identity(loaded.frame) + if actual_identity != stored_identity: + raise ValueError( + f"ASEC pre-clone checkpoint {checkpoint_path} Frame identity changed." + ) + _validate_asec_frame(loaded.frame, path=checkpoint_path) + metadata["identity"] = stored_identity.to_payload() + return loaded.frame, metadata + + +def _validate_outer_stage_binding( + metadata: dict[str, object], + *, + path: Path, +) -> dict[str, object]: + actual_keys = frozenset(metadata) + if actual_keys != _BINDING_KEYS: + raise ValueError( + f"ASEC pre-clone checkpoint {path} has an incomplete outer-stage " + f"artifact binding (missing: {sorted(_BINDING_KEYS - actual_keys)}; " + f"extra: {sorted(actual_keys - _BINDING_KEYS)})." + ) + if metadata["artifact_kind"] != _OUTER_STAGE_ARTIFACT_KIND: + raise ValueError( + f"ASEC pre-clone checkpoint {path} is not an outer-stage Frame artifact." + ) + if metadata["schema_version"] != OUTER_STAGE_CONTEXT_SCHEMA_VERSION or isinstance( + metadata["schema_version"], bool + ): + raise ValueError( + f"ASEC pre-clone checkpoint {path} has an unsupported outer-stage " + "schema version." + ) + if metadata["stage"] != _PRE_CLONE_STAGE: + raise ValueError( + f"ASEC pre-clone checkpoint {path} must be bound to stage " + f"{_PRE_CLONE_STAGE!r}, got {metadata['stage']!r}." + ) + if metadata["stage_index"] != _PRE_CLONE_STAGE_INDEX or isinstance( + metadata["stage_index"], bool + ): + raise ValueError( + f"ASEC pre-clone checkpoint {path} must be bound to stage_index " + f"{_PRE_CLONE_STAGE_INDEX}, got {metadata['stage_index']!r}." + ) + pipeline_sha256 = metadata["pipeline_sha256"] + if not isinstance(pipeline_sha256, str) or not _LOWERCASE_SHA256.fullmatch( + pipeline_sha256 + ): + raise ValueError( + f"ASEC pre-clone checkpoint {path} pipeline_sha256 must be a " + "lowercase SHA-256 digest." + ) + return dict(metadata) + + +def _validate_asec_frame(frame: Frame, *, path: Path) -> None: + if frame.schema != US_SCHEMA: + raise ValueError( + f"ASEC pre-clone checkpoint {path} must use the US entity schema." + ) + if frame.weighted_entities != ("household",): + raise ValueError( + f"ASEC pre-clone checkpoint {path} must carry household weights only; " + f"got weighted entities {list(frame.weighted_entities)}." + ) + weights = frame.weights_for("household") + if not isinstance(weights, Weights) or not isinstance(weights.kind, WeightKind): + raise ValueError( + f"ASEC pre-clone checkpoint {path} household weights must be typed." + ) + values = weights.values + if not np.isfinite(values).all() or not (values > 0.0).all(): + raise ValueError( + f"ASEC pre-clone checkpoint {path} household weights must be " + "strictly positive and finite." + ) diff --git a/packages/populace-build/tests/test_us_asec_checkpoint.py b/packages/populace-build/tests/test_us_asec_checkpoint.py new file mode 100644 index 00000000..aa5e0e53 --- /dev/null +++ b/packages/populace-build/tests/test_us_asec_checkpoint.py @@ -0,0 +1,194 @@ +from __future__ import annotations + +import json +from pathlib import Path + +import numpy as np +import pandas as pd +import pytest + +from populace.build.frame_checkpoint import write_frame_checkpoint +from populace.build.outer_stage_runtime import ( + OUTER_STAGE_CONTEXT_SCHEMA_VERSION, + frame_identity, +) +from populace.build.us_runtime import load_asec_pre_clone_checkpoint +from populace.frame import US_SCHEMA, EntitySchema, Frame, WeightKind, Weights + +_OUTER_STAGE_ARTIFACT_KIND = "populace_outer_stage_frame" + + +def _us_frame( + *, + id_offset: int = 0, + household_weights: tuple[float, float] = (2.0, 3.0), + person_weights: bool = False, +) -> Frame: + ids = np.asarray([1, 2], dtype=np.int64) + id_offset + person = pd.DataFrame( + { + "person_id": ids, + "person_household_id": ids, + "person_tax_unit_id": ids + 10, + "person_spm_unit_id": ids + 20, + "person_family_id": ids + 30, + "person_marital_unit_id": ids + 40, + "age": np.asarray([30, 50], dtype=np.int16), + } + ) + tables = { + "person": person, + "household": pd.DataFrame({"household_id": ids}), + "tax_unit": pd.DataFrame({"tax_unit_id": ids + 10}), + "spm_unit": pd.DataFrame({"spm_unit_id": ids + 20}), + "family": pd.DataFrame({"family_id": ids + 30}), + "marital_unit": pd.DataFrame({"marital_unit_id": ids + 40}), + } + weights = { + "household": Weights( + np.asarray(household_weights, dtype=np.float64), + WeightKind.DESIGN, + ) + } + if person_weights: + weights["person"] = Weights( + np.asarray([2.0, 3.0], dtype=np.float64), + WeightKind.DESIGN, + ) + return Frame( + tables, + US_SCHEMA, + weights, + pd.Series(["asec_2023", "asec_2024"], dtype=object), + ) + + +def _non_us_frame() -> Frame: + schema = EntitySchema(group_entities=("household",)) + return Frame( + { + "person": pd.DataFrame( + { + "person_id": np.asarray([1], dtype=np.int64), + "person_household_id": np.asarray([1], dtype=np.int64), + } + ), + "household": pd.DataFrame( + {"household_id": np.asarray([1], dtype=np.int64)} + ), + }, + schema, + { + "household": Weights( + np.asarray([1.0], dtype=np.float64), + WeightKind.DESIGN, + ) + }, + pd.Series(["fixture"], dtype=object), + ) + + +def _binding(frame: Frame) -> dict[str, object]: + return { + "artifact_kind": _OUTER_STAGE_ARTIFACT_KIND, + "identity": frame_identity(frame).to_payload(), + "pipeline_sha256": "a" * 64, + "schema_version": OUTER_STAGE_CONTEXT_SCHEMA_VERSION, + "stage": "pre_clone_enrichment", + "stage_index": 1, + } + + +def _write_checkpoint( + path: Path, + frame: Frame, + *, + metadata: dict[str, object] | None = None, +) -> None: + write_frame_checkpoint(path, frame, metadata=metadata or _binding(frame)) + + +def test_loads_bound_input_complete_asec_checkpoint(tmp_path: Path) -> None: + path = tmp_path / "001_pre_clone_enrichment.frame.h5" + source = _us_frame() + expected_metadata = _binding(source) + _write_checkpoint(path, source, metadata=expected_metadata) + + frame, metadata = load_asec_pre_clone_checkpoint(path) + + assert frame_identity(frame) == frame_identity(source) + assert frame.schema == US_SCHEMA + assert frame.weighted_entities == ("household",) + assert metadata == expected_metadata + assert json.loads(json.dumps(metadata)) == metadata + + +@pytest.mark.parametrize( + ("field", "value", "message"), + ( + ("artifact_kind", "other", "not an outer-stage Frame artifact"), + ("schema_version", 999, "unsupported outer-stage schema version"), + ("stage", "source_construction", "must be bound to stage"), + ("stage_index", 0, "must be bound to stage_index 1"), + ("pipeline_sha256", "not-a-digest", "lowercase SHA-256 digest"), + ), +) +def test_rejects_wrong_outer_stage_binding( + tmp_path: Path, + field: str, + value: object, + message: str, +) -> None: + path = tmp_path / f"wrong-{field}.frame.h5" + frame = _us_frame() + metadata = _binding(frame) + metadata[field] = value + _write_checkpoint(path, frame, metadata=metadata) + + with pytest.raises(ValueError, match=message): + load_asec_pre_clone_checkpoint(path) + + +def test_rejects_incomplete_outer_stage_binding(tmp_path: Path) -> None: + path = tmp_path / "missing-stage.frame.h5" + frame = _us_frame() + metadata = _binding(frame) + del metadata["stage"] + _write_checkpoint(path, frame, metadata=metadata) + + with pytest.raises(ValueError, match="incomplete outer-stage artifact binding"): + load_asec_pre_clone_checkpoint(path) + + +def test_rejects_identity_not_bound_to_loaded_frame(tmp_path: Path) -> None: + path = tmp_path / "wrong-identity.frame.h5" + frame = _us_frame() + metadata = _binding(frame) + metadata["identity"] = frame_identity(_us_frame(id_offset=100)).to_payload() + _write_checkpoint(path, frame, metadata=metadata) + + with pytest.raises(ValueError, match="Frame identity changed"): + load_asec_pre_clone_checkpoint(path) + + +@pytest.mark.parametrize( + ("frame", "message"), + ( + (_non_us_frame(), "must use the US entity schema"), + (_us_frame(person_weights=True), "must carry household weights only"), + ( + _us_frame(household_weights=(2.0, 0.0)), + "household weights must be strictly positive and finite", + ), + ), +) +def test_rejects_invalid_asec_frame_boundary( + tmp_path: Path, + frame: Frame, + message: str, +) -> None: + path = tmp_path / f"invalid-{len(list(tmp_path.iterdir()))}.frame.h5" + _write_checkpoint(path, frame) + + with pytest.raises(ValueError, match=message): + load_asec_pre_clone_checkpoint(path) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index eec8157c..51e6ea05 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -95,6 +95,7 @@ "acs_multispine.py", "acs_pums.py", "acs_sources.py", + "asec_checkpoint.py", # Bounded checkpoint I/O; no population treatment. "asec_pool.py", "base_pool.py", "block_ladder_sources.py", From 18ef73e908435fd82a033e67f25a05e7bede421a Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 20:42:23 -0400 Subject: [PATCH 09/94] Receipt disposable simulation defaults --- PROGRESS.md | 16 +++-- .../build/us_runtime/multispine_pool.py | 61 ++++++++++++++++++- .../tests/test_us_multispine_pool.py | 46 ++++++++++++++ 3 files changed, 115 insertions(+), 8 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 30b39702..93b77bbc 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -55,14 +55,18 @@ - Added fixed-batch SSI materialization on an ephemeral, receipt-preserving gate view. Formula-owned `ssi` is deliberately absent from the returned input pool. +- Added a disposable simulation projection that fills any still-null engine + inputs from the live engine defaults solely while materializing SSI. The + nullable pool remains untouched, and every temporary fill is receipted. +- Added the bounded ASEC pre-clone checkpoint loader: it requires the exact + `pre_clone_enrichment` stage binding, revalidates the stored frame identity, + and rejects non-US or invalid household-weight artifacts. ## Next -1. Finish missing-cell/raw-preserving transfer and receipt-safe production - operators. -2. Add the sha-pinned CLI with deterministic +1. Add the sha-pinned CLI with deterministic manifest/diagnostic paths and no tolerance knobs. -3. Convert the legacy CLI to a deprecated shim while preserving the helper - imports its known consumer needs. -4. Add synthetic full-path tests, changelog fragment, focused verification, and +2. Wire the raw-preserving transfer and receipt-safe production operators into + that CLI. +3. Add synthetic full-path tests, changelog fragment, focused verification, and the external review worklog. diff --git a/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py b/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py index 4e1281fa..4cf57e70 100644 --- a/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py +++ b/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py @@ -83,6 +83,10 @@ class _PoolRulesEngine(Protocol): def default_values(self, names: list[str]) -> Mapping[str, object]: ... + def variable_metadata(self, name: str) -> object: ... + + def variables(self) -> list[str]: ... + def materialize( self, bundle: Frame, @@ -293,7 +297,8 @@ def materialize_multispine_agreement_outputs( rules_engine = PolicyEngineUSEngine() - household_ids = frame.table("household")["household_id"].to_numpy() + simulation_frame, default_fills = _simulation_projection(frame, rules_engine) + household_ids = simulation_frame.table("household")["household_id"].to_numpy() person = frame.table("person") membership = person["person_household_id"] person_ids = person["person_id"] @@ -311,7 +316,7 @@ def materialize_multispine_agreement_outputs( low : low + POOL_SIMULATION_HOUSEHOLD_BATCH_SIZE ] person_mask = membership.isin(selected_households).to_numpy() - selected = frame.select(person_mask) + selected = simulation_frame.select(person_mask) materialized = np.asarray( rules_engine.materialize( selected, @@ -354,11 +359,63 @@ def materialize_multispine_agreement_outputs( }, "household_batch_size": POOL_SIMULATION_HOUSEHOLD_BATCH_SIZE, "batches": batch_count, + "simulation_projection_default_fills": default_fills, "persisted_to_pool": False, }, ) +def _simulation_projection( + frame: Frame, + engine: _PoolRulesEngine, +) -> tuple[Frame, dict[str, dict[str, object]]]: + """Fill nullable engine inputs only on the disposable simulation copy.""" + + variables_method = getattr(engine, "variables", None) + metadata_method = getattr(engine, "variable_metadata", None) + if not callable(variables_method) or not callable(metadata_method): + return frame, {} + + input_names = list(variables_method()) + defaults = dict(engine.default_values(input_names)) + tables = {entity: frame.table(entity).copy() for entity in frame.entities} + fills: dict[str, dict[str, object]] = {} + for name in input_names: + metadata = metadata_method(name) + entity = getattr(metadata, "entity", None) + if entity not in tables or name not in tables[entity]: + continue + missing = tables[entity][name].isna() + if not missing.any(): + continue + if name not in defaults: + raise ValueError( + f"SSI simulation projection cannot resolve {int(missing.sum())} " + f"missing value(s) in engine input {entity}.{name}; the engine " + "declares no default." + ) + tables[entity].loc[missing, name] = defaults[name] + fills[name] = { + "entity": entity, + "rows": int(missing.sum()), + "value": defaults[name], + "persisted_to_pool": False, + } + if not fills: + return frame, {} + return ( + Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ), + fills, + ) + + def _take_up_snapshots( frame: Frame, programs: tuple[TakeUpProgram, ...], diff --git a/packages/populace-build/tests/test_us_multispine_pool.py b/packages/populace-build/tests/test_us_multispine_pool.py index 50610ff0..09dbdf5a 100644 --- a/packages/populace-build/tests/test_us_multispine_pool.py +++ b/packages/populace-build/tests/test_us_multispine_pool.py @@ -1,6 +1,7 @@ from __future__ import annotations from collections.abc import Callable +from types import SimpleNamespace import numpy as np import pandas as pd @@ -371,3 +372,48 @@ def test_simulated_ssi_lives_only_on_receipt_preserving_gate_view() -> None: assert result.receipt["persisted_to_pool"] is False assert result.receipt["formula_outputs"]["ssi"]["rows"] == frame.n("person") assert sum(map(len, engine.materialized_person_ids)) == frame.n("person") + + +def test_simulation_defaults_are_disposable_and_receipted() -> None: + frame = seed_multispine_pool_inputs( + _assembled_cloned_with_partial_take_up(), + engine=_FakeEngine(), + ).frame + person = frame.table("person").copy() + person.loc[person.index[0], "age"] = np.nan + frame = _replace_person(frame, person) + + class ProjectionEngine(_FakeEngine): + def variables(self) -> list[str]: + return ["age"] + + def variable_metadata(self, name: str) -> object: + assert name == "age" + return SimpleNamespace(entity="person") + + def default_values(self, names: list[str]) -> dict[str, object]: + assert names == ["age"] + return {"age": 0.0} + + def materialize( + self, + bundle: Frame, + variables: list[str], + period: int, + ) -> dict[str, np.ndarray]: + assert not bundle.table("person")["age"].isna().any() + return super().materialize(bundle, variables, period) + + result = materialize_multispine_agreement_outputs( + frame, + engine=ProjectionEngine(), + ) + + assert result.frame.table("person")["age"].isna().sum() == 1 + assert result.frame.table("person").loc[person.index[0], "ssi"] == 0.0 + assert result.receipt["simulation_projection_default_fills"]["age"] == { + "entity": "person", + "rows": 1, + "value": 0.0, + "persisted_to_pool": False, + } From c47e216c1a324b2f5722752040fd7fa68593d9c8 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 20:48:35 -0400 Subject: [PATCH 10/94] Complete deterministic pool derivations --- .../build/us_runtime/multispine_pool.py | 129 ++++++++++++++++++ .../tests/test_us_multispine_pool.py | 29 ++++ 2 files changed, 158 insertions(+) diff --git a/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py b/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py index 4cf57e70..1971e572 100644 --- a/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py +++ b/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py @@ -23,8 +23,13 @@ from populace.build.us_runtime.acs_transfer import ( TargetFamilies, declared_acs_transfer_target_families, + derive_acs_schedule_d_capital_gain_distributions, ) from populace.build.us_runtime.puf_support import clone_us_frame_for_puf_support +from populace.build.us_runtime.qbi_inputs import ( + US_QBI_OUTPUT_COLUMNS, + with_us_qbi_input_reconciliation, +) from populace.build.us_runtime.spine_agreement import spine_agreement_gate from populace.build.us_runtime.spine_assembly import assemble_spines from populace.build.us_runtime.support_provenance import ( @@ -47,6 +52,7 @@ "POOL_TIME_PERIOD", "MultispinePoolResult", "PoolStageOutput", + "derive_multispine_pool_inputs", "materialize_multispine_agreement_outputs", "pool_transfer_target_families", "run_multispine_pool_path", @@ -149,6 +155,129 @@ def pool_transfer_target_families() -> TargetFamilies: } +def derive_multispine_pool_inputs(frame: Frame) -> PoolStageOutput: + """Complete deterministic post-transfer inputs without reading a spine. + + Schedule D capital-gain distributions are derived once per tax unit from + the transferred parent inputs, then carried by the first person only when + the unit has no pre-existing values. Existing non-null values are never + rewritten. The shared QBI reconciliation then restores its documented + all-or-nothing identities on the imputed PUF-detail surface. + """ + + with_schedule_d, schedule_d_receipt = _complete_schedule_d_input(frame) + reconciled = with_us_qbi_input_reconciliation(with_schedule_d) + return PoolStageOutput( + reconciled, + { + "schedule_d_capital_gain_distributions": schedule_d_receipt, + "qbi_input_reconciliation": { + "columns": list(US_QBI_OUTPUT_COLUMNS), + "operation": "shared_all_or_nothing_identity_reconciliation", + }, + }, + ) + + +def _complete_schedule_d_input(frame: Frame) -> tuple[Frame, dict[str, object]]: + person = frame.table("person") + membership_column = frame.schema.membership_column("tax_unit") + tax_unit_id_column = frame.schema.entity_id_column("tax_unit") + source_columns = ( + "long_term_capital_gains_before_response", + "non_sch_d_capital_gains", + ) + missing_sources = sorted( + column for column in source_columns if column not in person + ) + if missing_sources: + raise ValueError( + "Schedule D pool derivation requires transferred parent input(s): " + f"{missing_sources}." + ) + numeric_sources = person.loc[:, list(source_columns)].apply( + pd.to_numeric, + errors="coerce", + ) + source_values = numeric_sources.to_numpy(dtype=np.float64) + if not np.isfinite(source_values).all(): + raise ValueError( + "Schedule D pool derivation requires finite transferred parent inputs." + ) + + tax_unit_ids = frame.table("tax_unit")[tax_unit_id_column] + grouped = numeric_sources.groupby( + person[membership_column], + sort=False, + ).sum() + grouped = grouped.reindex(tax_unit_ids.to_numpy()) + if grouped.isna().any().any(): + raise ValueError( + "Schedule D pool derivation could not align every tax unit to people." + ) + derived, derivation_receipt = derive_acs_schedule_d_capital_gain_distributions( + grouped + ) + derived_by_tax_unit = dict(zip(tax_unit_ids.to_numpy(), derived, strict=True)) + + output_column = "schedule_d_capital_gain_distributions" + if output_column in person: + output = person[output_column].copy() + observed = output.notna() + observed_numeric = pd.to_numeric(output.loc[observed], errors="coerce") + if not np.isfinite(observed_numeric.to_numpy(dtype=np.float64)).all(): + raise ValueError( + "Schedule D pool derivation cannot preserve non-finite existing values." + ) + else: + output = pd.Series(np.nan, index=person.index, dtype=np.float64) + observed = pd.Series(False, index=person.index) + + derived_units = 0 + partially_observed_units = 0 + filled_rows = 0 + for tax_unit_id, row_indices in person.groupby( + membership_column, + sort=False, + ).groups.items(): + indices = list(row_indices) + missing = [index for index in indices if not bool(observed.loc[index])] + if not missing: + continue + if len(missing) == len(indices): + output.loc[missing[0]] = derived_by_tax_unit[tax_unit_id] + if len(missing) > 1: + output.loc[missing[1:]] = 0.0 + derived_units += 1 + else: + output.loc[missing] = 0.0 + partially_observed_units += 1 + filled_rows += len(missing) + + tables = {entity: frame.table(entity).copy() for entity in frame.entities} + tables["person"][output_column] = pd.to_numeric(output, errors="raise").to_numpy( + dtype=np.float64 + ) + completed = Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + return completed, { + "entity": "person", + "source_grain": "tax_unit", + "source_columns": list(source_columns), + "preserved_nonnull_rows": int(observed.sum()), + "filled_rows": filled_rows, + "derived_tax_units": derived_units, + "partially_observed_tax_units_filled_with_zero": partially_observed_units, + "derivation": derivation_receipt, + } + + def seed_multispine_pool_inputs( frame: Frame, *, diff --git a/packages/populace-build/tests/test_us_multispine_pool.py b/packages/populace-build/tests/test_us_multispine_pool.py index 09dbdf5a..b5c952fa 100644 --- a/packages/populace-build/tests/test_us_multispine_pool.py +++ b/packages/populace-build/tests/test_us_multispine_pool.py @@ -14,6 +14,7 @@ from populace.build.us_runtime.multispine_pool import ( POOL_OPERATOR_ORDER, PoolStageOutput, + _complete_schedule_d_input, materialize_multispine_agreement_outputs, pool_transfer_target_families, run_multispine_pool_path, @@ -264,6 +265,34 @@ def test_pool_transfer_plan_is_the_fixed_declared_qrf_surface() -> None: assert pool_transfer_target_families() == declared_acs_transfer_target_families() +def test_schedule_d_derivation_preserves_existing_values_and_receipt() -> None: + assembled = assemble_spines( + {"asec": _source_frame(), "acs": _source_frame()}, + household_mass_shares={"asec": 0.5, "acs": 0.5}, + ) + from populace.build.us_runtime.puf_support import ( + clone_us_frame_for_puf_support, + ) + + frame = clone_us_frame_for_puf_support(assembled) + person = frame.table("person").copy() + person["long_term_capital_gains_before_response"] = 100.0 + person["non_sch_d_capital_gains"] = 0.0 + person["schedule_d_capital_gain_distributions"] = np.nan + person.loc[person.index[0], "schedule_d_capital_gain_distributions"] = 7.0 + frame = _replace_person(frame, person) + + completed, receipt = _complete_schedule_d_input(frame) + + output = completed.table("person")["schedule_d_capital_gain_distributions"] + assert output.loc[person.index[0]] == 7.0 + assert not output.isna().any() + assert (output.loc[person.index[1:]] > 0.0).all() + assert completed.metadata == frame.metadata + assert receipt["preserved_nonnull_rows"] == 1 + assert receipt["filled_rows"] == len(person) - 1 + + class _FakeEngine: def __init__(self) -> None: self.materialized_person_ids: list[list[int]] = [] From f5d80b568fc10230cacf2dff9dc967da2ab42492 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 20:49:28 -0400 Subject: [PATCH 11/94] Document and guard multispine pool ordering --- .../578-multispine-pool-build.added.md | 1 + docs/us-multispine-operator-ordering.md | 116 ++++++++++++----- .../tests/test_us_spine_blindness.py | 123 ++++++++++++++++++ 3 files changed, 206 insertions(+), 34 deletions(-) create mode 100644 changelog.d/578-multispine-pool-build.added.md diff --git a/changelog.d/578-multispine-pool-build.added.md b/changelog.d/578-multispine-pool-build.added.md new file mode 100644 index 00000000..64e84e30 --- /dev/null +++ b/changelog.d/578-multispine-pool-build.added.md @@ -0,0 +1 @@ +Add a SHA-pinned, pre-calibration US multispine pool builder that assembles ASEC and ACS before PUF cloning and source-blind operators, emits nullable input-only artifacts with provenance receipts, and runs the fixed spine-agreement gate as its terminal stage; this code increment has synthetic fixture coverage but does not certify a full-data build. diff --git a/docs/us-multispine-operator-ordering.md b/docs/us-multispine-operator-ordering.md index e851ba1b..0e0742c5 100644 --- a/docs/us-multispine-operator-ordering.md +++ b/docs/us-multispine-operator-ordering.md @@ -1,16 +1,16 @@ # US multispine operator ordering -This note records the build ordering visible in -`tools/build_us_puf_support_base.py`, -`tools/build_us_acs_multispine_base.py`, and their directly called US runtime -modules. It defines the seam introduced by populace#395; it does not certify -the distributions of an output artifact. +This note records the executable pool ordering in +`tools/build_us_multispine_pool.py` and the serial lineage it replaces. The +assembly and agreement contracts originated in populace#581; populace#578 +increment 2 wires them into a build path. This documentation and its fixture +tests do not certify a full-data output artifact. -## Current ordering +## Retired serial ordering -The current lineage is two serial builds. The first build produces an -operated ASEC-by-PUF-detail donor. The second build creates ACS records, -transfers inputs from that donor, and only then appends ACS. +The earlier lineage used two serial builds. The first build produced an +operated ASEC-by-PUF-detail donor. The second build created ACS records, +transferred inputs from that donor, and only then appended ACS. ### `build_us_puf_support_base.py` @@ -35,8 +35,8 @@ stages. ### `build_us_acs_multispine_base.py` -The ACS builder takes that exported H5 as `--base-h5`. Its runtime call graph -is: +The former ACS builder took that exported H5 as `--base-h5`. Its runtime call +graph was: 1. Load and validate the dense ASEC-by-PUF-detail base and declared transfer coverage. @@ -55,14 +55,62 @@ is: Calibration is downstream of this tool. Simulation is not run by either builder in this call graph. -The important ordering fact is structural: ACS receives model-input +The important ordering flaw is structural: ACS receives model-input transfers from a donor after the donor has crossed the ASEC-only operator sequence. Appending the transferred ACS records later does not cause those operators to run over the combined population. -## Current provenance axes - -The current lineage uses two related metadata schemes: +`build_us_acs_multispine_base.py` is now a deprecated compatibility shim for +shared legacy H5 helpers. It does not expose a second executable pool builder. + +## Executable increment-2 pool build + +`build_us_multispine_pool.py` consumes only explicit local files and their +declared SHA-256 values: + +- the input-complete ASEC checkpoint from the + `pre_clone_enrichment` outer-stage boundary; +- the ACS household and person PUMS archives, whose caller-supplied hashes + must also match the checked-in ACS source manifest; and +- the processed PUF H5 and source-year PUF CSV used by the existing donor + loader. + +The tool does not download any source. It verifies all file pins before +loading frames, maps measured ACS fields without overwriting them, and then +runs this fixed sequence: + +1. `assemble_spines({"asec": ..., "acs": ...})` creates the first shared + population state and binds the immutable assembly receipt. +2. `clone_us_frame_for_puf_support(...)` applies the PUF-detail clone to the + whole assembled pool. Clone-index provenance, not source-spine identity, + controls later PUF-detail routing. +3. The primary PUF QRF chain and capital-gains tail transfer run over the + combined frame, followed by the declared ACS input-family QRF transfers. + Existing measured target cells remain unchanged, and transfer receipts + record fitted and imputed rows. +4. Deterministic input reconciliation runs over that same pool. +5. The seed stage preserves existing take-up values, applies the sourced + TANF and EITC mechanisms, and explicitly receipts live engine defaults + used for unresolved, non-transfer-owned take-up inputs. Those defaults + are not described as fitted or administrative mechanisms. +6. SSI is materialized only on an ephemeral agreement view in fixed + household batches. Any engine defaults required solely for that + calculation are separately receipted; formula output is not written into + the input pool. +7. The unchanged spine-agreement gate is terminal. It uses its checked-in + registry and fixed tolerances, batches all failures, and controls the + manifest's simulation-ready status. + +The output H5 is a nullable, input-only, pre-calibration pool. Its companion +manifest carries input pins, the assembly receipt, per-source and per-clone +counts, operator receipts, and the complete agreement result. A failed gate +writes diagnostics and a non-ready manifest and exits nonzero. Calibration +is deliberately absent; the downstream k-ladder may consume only a pool +whose terminal agreement result passed. + +## Provenance axes + +The retired lineage used two related metadata schemes: - PUF support cloning adds, on every entity, `*_source_id`, `*_support_channel`, and @@ -77,13 +125,13 @@ The current lineage uses two related metadata schemes: including the target family, donor spine/channel, predictors, seeds, weight kind, recipient patterns, and unmodeled-row count. -Those fields currently mix two concepts: the population source that carried -a record and the PUF-detail copy created by an operator. The new seam keeps -those concepts separate. +Those fields mixed two concepts: the population source that carried a record +and the PUF-detail copy created by an operator. The assembly seam keeps those +concepts separate. -## Canonical target ordering +## Canonical executable ordering -The target US multispine build order is: +The US multispine build order is: ```text source ingestion and faithful schema harmonization @@ -94,9 +142,11 @@ source ingestion and faithful schema harmonization -> seed take-up and other stochastic inputs -> simulate -> spine-agreement gate - -> calibrate + -> emit input-only pool and receipts ``` +Calibration is a downstream consumer boundary, not a stage in this tool. + `assemble_spines(...)` is the boundary between source preparation and population operators. It receives nullable, schema-compatible peer frames and produces one combined frame before cloning, fitted transfer, derivation, @@ -149,8 +199,11 @@ marital-unit, and benefit-unit naming variants. It recognizes direct, aliased-helper, dynamic-subscript, `getattr`, and `*_spine_source_id` reads. Every US runtime module is classified as a reviewed population operator or an explicit non-operator/provenance owner, so a new unclassified module fails the -guard. `transfer_acs_inputs` selects fit donors by the centrally derived clone -role; assembled source-channel names never determine donor eligibility. +guard. A second fail-closed scan starts at `build_us_multispine_pool.py` and +covers the tool plus its transitive US-runtime import graph under the same +owner registry. `transfer_acs_inputs` selects fit donors by the centrally +derived clone role; assembled source-channel names never determine donor +eligibility. Assembly accepts only integer-typed, nonnegative structural source IDs and names the source spine and offending IDs on failure. PUF cloning revalidates @@ -217,15 +270,10 @@ This ordering makes the gate diagnostic of the shared operator surface: calibration cannot hide a disagreement, and no per-spine target, loss term, seed, or tolerance may be introduced to shape a passing result. -## Increment-1 compatibility boundary - -This increment adds an opt-in assembly seam, operator contracts, structural -enforcement, and the agreement-gate specification. It does not rewire -`build_us_puf_support_base.py`, -`build_us_acs_multispine_base.py`, or current sparse/dense release tools to -the new sequence. Their current call paths and artifact behavior remain the -compatibility lineage until a later increment explicitly adopts the seam. +## Increment-2 validation boundary -The seam is the foundation for the broader populace#578 build shape, -including one suite per country and the US full-geography/exact-k work. Those -release changes are outside this increment. +This increment makes the assembly seam executable and covers it with small, +synthetic two-spine fixtures. It does not execute or certify a full-data +build, download a source dataset, calibrate the pool, select k, or change the +current sparse/dense release artifacts. Those data-scale and release steps +remain downstream of this code increment. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 51e6ea05..743fbaac 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -6,9 +6,20 @@ import fnmatch from pathlib import Path +_REPOSITORY_ROOT = Path(__file__).resolve().parents[3] _US_RUNTIME = ( Path(__file__).resolve().parents[1] / "src" / "populace" / "build" / "us_runtime" ) +_US_RUNTIME_IMPORT_PREFIX = "populace.build.us_runtime" +_SPINE_BLIND_BUILD_TOOLS = (_REPOSITORY_ROOT / "tools" / "build_us_multispine_pool.py",) +_REQUIRED_POOL_RUNTIME_MODULES = frozenset( + { + "multispine_pool.py", + "puf_support.py", + "spine_agreement.py", + "spine_assembly.py", + } +) # These modules own source-spine provenance rather than applying population # treatments. Keep the allowlist exact so adding a new exception requires a @@ -446,6 +457,63 @@ def _called_function_names(source: str) -> set[str]: } +def _imported_us_runtime_modules(source: str) -> tuple[str, ...]: + """Return statically imported, flat ``us_runtime`` module filenames.""" + + imported: set[str] = set() + for node in ast.walk(ast.parse(source)): + if isinstance(node, ast.Import): + module_names = (alias.name for alias in node.names) + elif isinstance(node, ast.ImportFrom) and node.module is not None: + if node.module == _US_RUNTIME_IMPORT_PREFIX: + module_names = ( + f"{_US_RUNTIME_IMPORT_PREFIX}.{alias.name}" + for alias in node.names + if (_US_RUNTIME / f"{alias.name}.py").is_file() + ) + else: + module_names = (node.module,) + else: + continue + + for module_name in module_names: + prefix = f"{_US_RUNTIME_IMPORT_PREFIX}." + if not module_name.startswith(prefix): + continue + relative = module_name.removeprefix(prefix) + leaf = relative.split(".", maxsplit=1)[0] + imported.add(f"{leaf}.py") + return tuple(sorted(imported)) + + +def _us_runtime_import_graph( + root: Path, +) -> tuple[tuple[Path, ...], tuple[str, ...]]: + """Resolve the runtime portion of one tool's static import graph.""" + + pending = list(_imported_us_runtime_modules(root.read_text())) + visited: set[str] = set() + missing: set[str] = set() + while pending: + module_name = pending.pop() + if module_name in visited: + continue + visited.add(module_name) + path = _US_RUNTIME / module_name + if not path.is_file(): + missing.add(module_name) + continue + pending.extend( + imported + for imported in _imported_us_runtime_modules(path.read_text()) + if imported not in visited + ) + return ( + tuple(_US_RUNTIME / name for name in sorted(visited - missing)), + tuple(sorted(missing)), + ) + + def _frame_metadata_drops(source: str) -> tuple[str, ...]: """Find Frame rebuilds that carry a mass log but drop stage metadata.""" @@ -577,6 +645,61 @@ def test_registered_population_operators_do_not_read_any_source_channel() -> Non ) +def test_pool_build_tool_import_graph_is_source_spine_blind() -> None: + """The wired CLI and every runtime operator it reaches remain blind.""" + + missing_tools = [ + str(path) for path in _SPINE_BLIND_BUILD_TOOLS if not path.is_file() + ] + assert not missing_tools, ( + f"registered spine-blind build tools are missing: {missing_tools}" + ) + + for tool in _SPINE_BLIND_BUILD_TOOLS: + runtime_graph, missing_modules = _us_runtime_import_graph(tool) + assert not missing_modules, ( + f"{tool.name} imports unresolved US runtime modules: {missing_modules}" + ) + runtime_names = {path.name for path in runtime_graph} + missing_required = sorted(_REQUIRED_POOL_RUNTIME_MODULES - runtime_names) + assert not missing_required, ( + f"{tool.name} does not reach the canonical pool seam modules: " + f"{missing_required}" + ) + unclassified = _unclassified_runtime_modules(runtime_names) + assert not unclassified, ( + f"{tool.name} reaches unclassified US runtime modules: {unclassified}" + ) + + offenders: dict[str, tuple[str, ...]] = {} + for path in (tool, *runtime_graph): + if path.name in _SOURCE_SPINE_PROVENANCE_OWNERS: + continue + reads = _source_spine_accesses(path.read_text()) + if reads: + offenders[str(path.relative_to(_REPOSITORY_ROOT))] = reads + assert not offenders, ( + "The multispine pool build path must remain source-spine blind. " + "Only the existing provenance-owner modules may inspect source " + f"identity; found: {offenders}" + ) + + +def test_pool_build_import_graph_parser_covers_supported_import_forms() -> None: + """Direct, aliased, and package-qualified runtime imports are resolved.""" + + source = """ +import populace.build.us_runtime.acs_transfer as transfer +from populace.build.us_runtime.multispine_pool import run_multispine_pool_path +from populace.build.us_runtime import puf_support +""" + assert _imported_us_runtime_modules(source) == ( + "acs_transfer.py", + "multispine_pool.py", + "puf_support.py", + ) + + def test_puf_clone_operators_resolve_clone_index_metadata() -> None: """The PUF clone, imputation, and tail stages route by clone index.""" From f42eaed65f554fc712134f839b1b53087971551a Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 20:50:30 -0400 Subject: [PATCH 12/94] Reject retired late assembly in pool graph --- .../populace-build/tests/test_us_spine_blindness.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 743fbaac..4337be6a 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -20,6 +20,12 @@ "spine_assembly.py", } ) +_RETIRED_LATE_ASSEMBLY_MODULES = frozenset( + { + "acs_multispine.py", + "base_pool.py", + } +) # These modules own source-spine provenance rather than applying population # treatments. Keep the allowlist exact so adding a new exception requires a @@ -666,6 +672,10 @@ def test_pool_build_tool_import_graph_is_source_spine_blind() -> None: f"{tool.name} does not reach the canonical pool seam modules: " f"{missing_required}" ) + retired_modules = sorted(_RETIRED_LATE_ASSEMBLY_MODULES & runtime_names) + assert not retired_modules, ( + f"{tool.name} reaches retired late-assembly modules: {retired_modules}" + ) unclassified = _unclassified_runtime_modules(runtime_names) assert not unclassified, ( f"{tool.name} reaches unclassified US runtime modules: {unclassified}" From 8e3b35af8daa57b9e0d00ab28349d9a858e4c9db Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 20:54:58 -0400 Subject: [PATCH 13/94] Wire SHA-pinned multispine pool builder --- PROGRESS.md | 22 +- tools/build_us_multispine_pool.py | 686 ++++++++++++++++++++++++++++++ 2 files changed, 703 insertions(+), 5 deletions(-) create mode 100644 tools/build_us_multispine_pool.py diff --git a/PROGRESS.md b/PROGRESS.md index 93b77bbc..e767de52 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -61,12 +61,24 @@ - Added the bounded ASEC pre-clone checkpoint loader: it requires the exact `pre_clone_enrichment` stage binding, revalidates the stored frame identity, and rejects non-US or invalid household-weight artifacts. +- Added an explicit spine-blind derivation stage that completes the Schedule D + memo input at tax-unit grain without rewriting existing values, then applies + the shared QBI identity reconciliation while preserving the assembly receipt. +- Added `tools/build_us_multispine_pool.py`: its CLI requires five explicit + path/SHA pairs plus `--out`, rechecks packaged ACS byte pins without + downloading, runs the fixed pool order, retains the resumable fixed-parameter + QRF checkpoint, and writes an input-only nullable H5, terminal agreement + diagnostics, and an authoritative manifest. Red agreement returns nonzero + with `simulation_ready=false`; calibration remains downstream. +- Extended the AST guard transitively from the new CLI through its US runtime + imports, with no new source-provenance owner exceptions, and rejected the + retired late-assembly graph. ## Next -1. Add the sha-pinned CLI with deterministic - manifest/diagnostic paths and no tolerance knobs. -2. Wire the raw-preserving transfer and receipt-safe production operators into - that CLI. -3. Add synthetic full-path tests, changelog fragment, focused verification, and +1. Add synthetic full-path CLI tests for red agreement receipts, manifest + contents, structural #581 errors, and pre-load SHA refusal. +2. Verify and fix the ACS/ASEC source-lineage dtype boundary discovered during + the production input audit. +3. Run focused verification, update the committed ledger, and write the external review worklog. diff --git a/tools/build_us_multispine_pool.py b/tools/build_us_multispine_pool.py new file mode 100644 index 00000000..1545f359 --- /dev/null +++ b/tools/build_us_multispine_pool.py @@ -0,0 +1,686 @@ +#!/usr/bin/env python3 +"""Build the SHA-pinned, pre-calibration US multispine input pool. + +The executable order is fixed: + +``assemble -> clone -> impute -> derive -> seed -> simulate -> agreement``. + +Every input is local and explicitly SHA-pinned; this tool never downloads +data. It writes a nullable input-only H5 plus a manifest and terminal agreement +diagnostics. A failed agreement gate still leaves those diagnostic artifacts +but returns nonzero and never marks the pool simulation-ready. Calibration is +deliberately downstream. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import os +import re +import uuid +from collections.abc import Callable, Mapping +from dataclasses import asdict, dataclass, is_dataclass +from enum import Enum +from pathlib import Path +from typing import Any + +import numpy as np +import pandas as pd + +from populace.build.gates import FitWeightRecord, GateReport, weights_audit_gate +from populace.build.us_runtime.acs_inputs import map_acs_native_inputs +from populace.build.us_runtime.acs_pums import ( + AcsPumsSource, + build_acs_pums_unit_frame, +) +from populace.build.us_runtime.acs_sources import ( + AcsSourceArtifact, + AcsSourceManifest, + load_acs_source_manifest, +) +from populace.build.us_runtime.acs_transfer import ( + ACS_DONOR_CHANNEL_AUTO, + DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, + transfer_acs_inputs, +) +from populace.build.us_runtime.asec_checkpoint import ( + load_asec_pre_clone_checkpoint, +) +from populace.build.us_runtime.h5_io import write_nullable_us_h5 +from populace.build.us_runtime.multispine_pool import ( + POOL_OPERATOR_ORDER, + POOL_RANDOM_SEED, + POOL_TIME_PERIOD, + MultispinePoolResult, + PoolStageOutput, + derive_multispine_pool_inputs, + materialize_multispine_agreement_outputs, + pool_transfer_target_families, + run_multispine_pool_path, + seed_multispine_pool_inputs, +) +from populace.build.us_runtime.puf_capital_gains_tail import ( + transfer_puf_capital_gains_tail, + validate_puf_capital_gains_tail_manifest, +) +from populace.build.us_runtime.puf_donor_io import load_puf_tax_unit_donor +from populace.build.us_runtime.puf_qrf_chain import ( + PRIMARY_QRF_MANIFEST_FILENAME, + finalize_primary_puf_qrf_chain, + initialize_primary_puf_qrf_chain, + run_primary_puf_qrf_chain, +) +from populace.build.us_runtime.puf_support import US_PUF_SUPPORT_FIT_NAME +from populace.frame import Frame + +__all__ = [ + "POOL_H5_ARTIFACT_KIND", + "POOL_MANIFEST_SCHEMA_VERSION", + "PoolBuildOutputs", + "build_multispine_pool", + "main", +] + +POOL_MANIFEST_SCHEMA_VERSION = 1 +"""Schema version for the companion pool build manifest.""" + +POOL_H5_ARTIFACT_KIND = "populace_us_multispine_input_pool" +"""Neutral H5 artifact kind; readiness is asserted only by the manifest.""" + +_PRIMARY_QRF_N_ESTIMATORS = 100 +_ACS_TRANSFER_N_ESTIMATORS = 100 +_LOWERCASE_SHA256 = re.compile(r"[0-9a-f]{64}") + +type PoolOperator = Callable[[Frame], PoolStageOutput] + + +@dataclass(frozen=True) +class PoolBuildOutputs: + """Deterministic output paths derived from the requested H5 path.""" + + pool_h5: Path + manifest: Path + agreement_diagnostics: Path + primary_qrf_checkpoint_dir: Path + + +@dataclass(frozen=True) +class _VerifiedInput: + role: str + path: Path + expected_sha256: str + actual_sha256: str + size_bytes: int + + def to_manifest(self) -> dict[str, object]: + return { + "path": str(self.path.resolve()), + "expected_sha256": self.expected_sha256, + "actual_sha256": self.actual_sha256, + "size_bytes": self.size_bytes, + } + + +@dataclass(frozen=True) +class _LoadedInputs: + asec: Frame + acs: Frame + puf_donor: pd.DataFrame + asec_checkpoint: Mapping[str, object] + acs_build: Mapping[str, object] + acs_native_inputs: Mapping[str, Mapping[str, Any]] + puf_donor_build: Mapping[str, object] + + +def _sha256_argument(value: str) -> str: + if not _LOWERCASE_SHA256.fullmatch(value): + raise argparse.ArgumentTypeError( + "SHA-256 pins must be exactly 64 lowercase hexadecimal characters." + ) + return value + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--asec-pre-clone-h5", + required=True, + type=Path, + help="Input-complete ASEC pre_clone_enrichment Frame checkpoint.", + ) + parser.add_argument( + "--asec-pre-clone-h5-sha256", + required=True, + type=_sha256_argument, + help="Expected SHA-256 of --asec-pre-clone-h5.", + ) + parser.add_argument( + "--acs-household-zip", + required=True, + type=Path, + help="Local packaged-pin ACS 2024 1-year household PUMS archive.", + ) + parser.add_argument( + "--acs-household-zip-sha256", + required=True, + type=_sha256_argument, + help="Expected SHA-256 of --acs-household-zip.", + ) + parser.add_argument( + "--acs-person-zip", + required=True, + type=Path, + help="Local packaged-pin ACS 2024 1-year person PUMS archive.", + ) + parser.add_argument( + "--acs-person-zip-sha256", + required=True, + type=_sha256_argument, + help="Expected SHA-256 of --acs-person-zip.", + ) + parser.add_argument( + "--puf-h5", + required=True, + type=Path, + help="Processed PUF tax-unit donor H5.", + ) + parser.add_argument( + "--puf-h5-sha256", + required=True, + type=_sha256_argument, + help="Expected SHA-256 of --puf-h5.", + ) + parser.add_argument( + "--puf-source-year-csv", + required=True, + type=Path, + help="Restricted source-year PUF CSV used for E00100 alignment.", + ) + parser.add_argument( + "--puf-source-year-csv-sha256", + required=True, + type=_sha256_argument, + help="Expected SHA-256 of --puf-source-year-csv.", + ) + parser.add_argument( + "--out", + required=True, + type=Path, + help="Destination nullable input-pool H5; sidecars derive from this path.", + ) + return parser + + +def _output_paths(path: Path) -> PoolBuildOutputs: + pool_h5 = Path(path) + if pool_h5.suffix.lower() not in {".h5", ".hdf5"}: + raise ValueError("--out must name an .h5 or .hdf5 file.") + return PoolBuildOutputs( + pool_h5=pool_h5, + manifest=pool_h5.with_suffix(".manifest.json"), + agreement_diagnostics=pool_h5.with_suffix(".agreement.json"), + primary_qrf_checkpoint_dir=pool_h5.with_suffix(".primary-qrf"), + ) + + +def _verify_file(role: str, path: Path, expected_sha256: str) -> _VerifiedInput: + source = Path(path) + if not source.is_file(): + raise FileNotFoundError(f"{role} input is not a file: {source}") + actual_sha256 = _file_sha256(source) + if actual_sha256 != expected_sha256: + raise ValueError( + f"{role} SHA-256 mismatch for {source}: got {actual_sha256}, " + f"expected {expected_sha256}." + ) + return _VerifiedInput( + role=role, + path=source, + expected_sha256=expected_sha256, + actual_sha256=actual_sha256, + size_bytes=source.stat().st_size, + ) + + +def _verify_acs_file( + role: str, + path: Path, + expected_sha256: str, + packaged: AcsSourceArtifact, +) -> _VerifiedInput: + if expected_sha256 != packaged.sha256: + raise ValueError( + f"{role} CLI pin differs from the packaged ACS source pin: " + f"got {expected_sha256}, expected {packaged.sha256}." + ) + verified = _verify_file(role, path, expected_sha256) + if verified.size_bytes != packaged.size_bytes: + raise ValueError( + f"{role} byte-size mismatch for {path}: got {verified.size_bytes}, " + f"expected packaged size {packaged.size_bytes}." + ) + return verified + + +def _verify_inputs( + args: argparse.Namespace, + outputs: PoolBuildOutputs, +) -> tuple[dict[str, _VerifiedInput], AcsSourceManifest]: + source_paths = { + Path(args.asec_pre_clone_h5).resolve(), + Path(args.acs_household_zip).resolve(), + Path(args.acs_person_zip).resolve(), + Path(args.puf_h5).resolve(), + Path(args.puf_source_year_csv).resolve(), + } + output_paths = { + outputs.pool_h5.resolve(), + outputs.manifest.resolve(), + outputs.agreement_diagnostics.resolve(), + outputs.primary_qrf_checkpoint_dir.resolve(), + } + collisions = sorted(str(path) for path in source_paths & output_paths) + if collisions: + raise ValueError(f"Pool outputs must not overwrite inputs: {collisions}.") + + acs_source_manifest = load_acs_source_manifest() + verified = { + "asec_pre_clone": _verify_file( + "ASEC pre-clone checkpoint", + args.asec_pre_clone_h5, + args.asec_pre_clone_h5_sha256, + ), + "acs_household": _verify_acs_file( + "ACS household archive", + args.acs_household_zip, + args.acs_household_zip_sha256, + acs_source_manifest.artifact("household"), + ), + "acs_person": _verify_acs_file( + "ACS person archive", + args.acs_person_zip, + args.acs_person_zip_sha256, + acs_source_manifest.artifact("person"), + ), + "processed_puf": _verify_file( + "processed PUF H5", + args.puf_h5, + args.puf_h5_sha256, + ), + "puf_source_year": _verify_file( + "source-year PUF CSV", + args.puf_source_year_csv, + args.puf_source_year_csv_sha256, + ), + } + return verified, acs_source_manifest + + +def _load_inputs( + args: argparse.Namespace, + *, + acs_source_manifest: AcsSourceManifest, +) -> _LoadedInputs: + asec, asec_checkpoint = load_asec_pre_clone_checkpoint(args.asec_pre_clone_h5) + acs_source = AcsPumsSource( + household_zip=args.acs_household_zip, + person_zip=args.acs_person_zip, + vintage=acs_source_manifest.vintage, + ) + acs_frame, acs_build = build_acs_pums_unit_frame(acs_source) + mapped_acs = map_acs_native_inputs(acs_frame) + donor_build: dict[str, object] = {} + puf_donor = load_puf_tax_unit_donor( + args.puf_h5, + args.puf_source_year_csv, + donor_build_summary=donor_build, + ) + return _LoadedInputs( + asec=asec, + acs=mapped_acs.frame, + puf_donor=puf_donor, + asec_checkpoint=asec_checkpoint, + acs_build=acs_build, + acs_native_inputs=mapped_acs.native_inputs, + puf_donor_build=donor_build, + ) + + +def _primary_qrf_manifest_path(checkpoint_dir: Path) -> Path: + return checkpoint_dir / PRIMARY_QRF_MANIFEST_FILENAME + + +def _initialize_or_resume_primary_qrf( + frame: Frame, + donor: pd.DataFrame, + checkpoint_dir: Path, +) -> None: + manifest_path = _primary_qrf_manifest_path(checkpoint_dir) + if manifest_path.is_file(): + return + if checkpoint_dir.exists(): + if not checkpoint_dir.is_dir(): + raise ValueError( + "Primary QRF checkpoint path exists but is not a directory: " + f"{checkpoint_dir}." + ) + if any(checkpoint_dir.iterdir()): + raise ValueError( + "Primary QRF checkpoint directory is nonempty but has no bound " + f"manifest: {checkpoint_dir}." + ) + initialize_primary_puf_qrf_chain( + frame, + donor, + checkpoint_dir, + seed=POOL_RANDOM_SEED, + n_estimators=_PRIMARY_QRF_N_ESTIMATORS, + ) + + +def _impute_pool( + frame: Frame, + *, + puf_donor: pd.DataFrame, + checkpoint_dir: Path, +) -> PoolStageOutput: + _initialize_or_resume_primary_qrf(frame, puf_donor, checkpoint_dir) + run_primary_puf_qrf_chain(checkpoint_dir) + + tail_bound_diagnostics: list[dict[str, object]] = [] + with_primary_detail, primary_weight_kind = finalize_primary_puf_qrf_chain( + frame, + checkpoint_dir, + tail_bound_diagnostics=tail_bound_diagnostics, + ) + with_tail, tail_receipt = transfer_puf_capital_gains_tail( + with_primary_detail, + puf_donor, + seed=POOL_RANDOM_SEED, + ) + validate_puf_capital_gains_tail_manifest(tail_receipt) + tail_ceiling = tail_receipt["tail_distribution_receipts"]["frame_after_stage"] + if not tail_ceiling["positive_mass_five_x_target_exceeded"]: + raise ValueError( + "PUF capital-gains tail transfer did not clear its declared " + "five-times positive-mass target: " + f"{tail_ceiling['positive_mass_five_x_ceiling']} <= " + f"{tail_ceiling['positive_mass_five_x_target']}." + ) + transfer_families = pool_transfer_target_families() + transferred = transfer_acs_inputs( + with_tail, + with_tail, + target_families=transfer_families, + donor_channel=ACS_DONOR_CHANNEL_AUTO, + seed=POOL_RANDOM_SEED, + n_estimators=_ACS_TRANSFER_N_ESTIMATORS, + max_targets_per_fit=DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, + ) + + fit_records = ( + FitWeightRecord(US_PUF_SUPPORT_FIT_NAME, primary_weight_kind), + *transferred.fit_records, + ) + weights_audit = weights_audit_gate(fit_records) + if not weights_audit.passed: + raise ValueError( + "Pool imputation weights audit failed:\n " + + "\n ".join(weights_audit.failures) + ) + + qrf_manifest_path = _primary_qrf_manifest_path(checkpoint_dir) + return PoolStageOutput( + transferred.frame, + { + "primary_puf_qrf": { + "checkpoint_manifest": _read_json_object(qrf_manifest_path), + "checkpoint_manifest_sha256": _file_sha256(qrf_manifest_path), + "n_estimators": _PRIMARY_QRF_N_ESTIMATORS, + "tail_bound_diagnostics": tail_bound_diagnostics, + }, + "puf_capital_gains_tail_transfer": tail_receipt, + "acs_qrf_transfer": { + "target_families": transfer_families, + "n_estimators": _ACS_TRANSFER_N_ESTIMATORS, + "max_targets_per_fit": DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, + "resolved_donor_channel": transferred.resolved_donor_channel, + "imputed_inputs": list(transferred.imputed_inputs), + "fit_records": list(transferred.fit_records), + "deferred_inputs": list(transferred.deferred_inputs), + }, + "weights_audit": GateReport((weights_audit,)).to_manifest(), + }, + ) + + +def build_multispine_pool( + asec: Frame, + acs: Frame, + *, + puf_donor: pd.DataFrame, + primary_qrf_checkpoint_dir: Path, + impute: PoolOperator | None = None, + derive: PoolOperator = derive_multispine_pool_inputs, + seed: PoolOperator = seed_multispine_pool_inputs, + simulate: PoolOperator = materialize_multispine_agreement_outputs, +) -> MultispinePoolResult: + """Run the production pool path, with injectable operators for fixtures. + + Production callers omit all operator overrides. The terminal agreement + gate is intentionally not injectable here, so this wiring can never alter + its registry or fixed tolerances. + """ + + impute_operator = ( + ( + lambda frame: _impute_pool( + frame, + puf_donor=puf_donor, + checkpoint_dir=primary_qrf_checkpoint_dir, + ) + ) + if impute is None + else impute + ) + return run_multispine_pool_path( + asec, + acs, + impute=impute_operator, + derive=derive, + seed=seed, + simulate=simulate, + ) + + +def _agreement_payload(result: MultispinePoolResult) -> dict[str, object]: + return GateReport((result.agreement_gate,)).to_manifest() + + +def _manifest_payload( + *, + result: MultispinePoolResult, + outputs: PoolBuildOutputs, + verified_inputs: Mapping[str, _VerifiedInput], + acs_source_manifest: AcsSourceManifest, + loaded: _LoadedInputs, +) -> dict[str, object]: + status = "simulation_ready" if result.simulation_ready else "agreement_failed" + return { + "artifact_kind": "populace_us_multispine_pool_manifest", + "schema_version": POOL_MANIFEST_SCHEMA_VERSION, + "status": status, + "simulation_ready": result.simulation_ready, + "calibration_applied": False, + "operator_order": list(POOL_OPERATOR_ORDER), + "period": POOL_TIME_PERIOD, + "random_seed": POOL_RANDOM_SEED, + "provenance_pins": { + role: pin.to_manifest() for role, pin in verified_inputs.items() + }, + "asec_pre_clone_checkpoint": loaded.asec_checkpoint, + "acs_source_manifest": asdict(acs_source_manifest), + "acs_pums_build": loaded.acs_build, + "acs_native_inputs": loaded.acs_native_inputs, + "puf_donor": { + "rows": int(len(loaded.puf_donor)), + "columns": sorted(str(column) for column in loaded.puf_donor.columns), + "build_receipt": loaded.puf_donor_build, + }, + "assembly_receipt": result.assembly_receipt, + "provenance_counts": result.provenance_counts, + "stage_receipts": result.stage_receipts, + "agreement_gate": _agreement_payload(result), + "pool_h5": { + "path": str(outputs.pool_h5.resolve()), + "sha256": _file_sha256(outputs.pool_h5), + "size_bytes": outputs.pool_h5.stat().st_size, + "artifact_kind": POOL_H5_ARTIFACT_KIND, + "nullable": True, + "input_only": True, + "formula_outputs_persisted": False, + }, + "agreement_diagnostics": { + "path": str(outputs.agreement_diagnostics.resolve()), + "sha256": _file_sha256(outputs.agreement_diagnostics), + "size_bytes": outputs.agreement_diagnostics.stat().st_size, + }, + "primary_qrf_checkpoint_dir": str(outputs.primary_qrf_checkpoint_dir.resolve()), + "calibration": { + "applied": False, + "position": "downstream", + "consumer": "k-ladder", + "requires_manifest_simulation_ready": True, + }, + } + + +def _write_outputs( + result: MultispinePoolResult, + *, + outputs: PoolBuildOutputs, + verified_inputs: Mapping[str, _VerifiedInput], + acs_source_manifest: AcsSourceManifest, + loaded: _LoadedInputs, +) -> None: + write_nullable_us_h5( + result.frame, + outputs.pool_h5, + period=POOL_TIME_PERIOD, + artifact_kind=POOL_H5_ARTIFACT_KIND, + ) + diagnostics = { + "artifact_kind": "populace_us_multispine_agreement_diagnostics", + "schema_version": POOL_MANIFEST_SCHEMA_VERSION, + "simulation_ready": result.simulation_ready, + "agreement_gate": _agreement_payload(result), + } + _atomic_write_json(outputs.agreement_diagnostics, diagnostics) + manifest = _manifest_payload( + result=result, + outputs=outputs, + verified_inputs=verified_inputs, + acs_source_manifest=acs_source_manifest, + loaded=loaded, + ) + _atomic_write_json(outputs.manifest, manifest) + + +def _file_sha256(path: Path) -> str: + digest = hashlib.sha256() + with Path(path).open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _read_json_object(path: Path) -> dict[str, object]: + raw = json.loads(Path(path).read_text(encoding="utf-8")) + if not isinstance(raw, dict): + raise ValueError(f"Expected a JSON object at {path}.") + return raw + + +def _json_ready(value: object) -> object: + if is_dataclass(value) and not isinstance(value, type): + return _json_ready(asdict(value)) + if isinstance(value, Mapping): + if any(not isinstance(key, str) for key in value): + raise ValueError("Build receipt mappings must use string JSON keys.") + return {key: _json_ready(item) for key, item in value.items()} + if isinstance(value, (list, tuple)): + return [_json_ready(item) for item in value] + if isinstance(value, np.ndarray): + return [_json_ready(item) for item in value.tolist()] + if isinstance(value, np.generic): + return _json_ready(value.item()) + if isinstance(value, Path): + return str(value) + if isinstance(value, Enum): + return _json_ready(value.value) + if isinstance(value, float) and not math.isfinite(value): + raise ValueError("Build receipts must not contain non-finite JSON numbers.") + if value is pd.NA: + return None + return value + + +def _atomic_write_json(path: Path, payload: Mapping[str, object]) -> None: + output = Path(path) + output.parent.mkdir(parents=True, exist_ok=True) + temporary = output.with_name(f".{output.name}.{uuid.uuid4().hex}.tmp") + try: + with temporary.open("w", encoding="utf-8") as stream: + stream.write( + json.dumps( + _json_ready(payload), + allow_nan=False, + indent=2, + sort_keys=True, + ) + + "\n" + ) + stream.flush() + os.fsync(stream.fileno()) + os.replace(temporary, output) + finally: + temporary.unlink(missing_ok=True) + + +def main(argv: list[str] | None = None) -> int: + """Build the pool and return nonzero exactly when terminal agreement is red.""" + + args = _parser().parse_args(argv) + outputs = _output_paths(args.out) + verified_inputs, acs_source_manifest = _verify_inputs(args, outputs) + loaded = _load_inputs(args, acs_source_manifest=acs_source_manifest) + result = build_multispine_pool( + loaded.asec, + loaded.acs, + puf_donor=loaded.puf_donor, + primary_qrf_checkpoint_dir=outputs.primary_qrf_checkpoint_dir, + ) + _write_outputs( + result, + outputs=outputs, + verified_inputs=verified_inputs, + acs_source_manifest=acs_source_manifest, + loaded=loaded, + ) + if not result.simulation_ready: + print( + "US multispine agreement failed; diagnostics and a non-ready " + f"manifest were written to {outputs.agreement_diagnostics} and " + f"{outputs.manifest}.", + ) + return 1 + print(f"Wrote simulation-ready multispine pool: {outputs.pool_h5}") + print(f"Wrote pool manifest: {outputs.manifest}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 5e4f560f8e7fd48c527702262e9ca771c1d82080 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 20:57:03 -0400 Subject: [PATCH 14/94] Bind pool assembly and tail receipts --- tools/build_us_multispine_pool.py | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/tools/build_us_multispine_pool.py b/tools/build_us_multispine_pool.py index 1545f359..220f3012 100644 --- a/tools/build_us_multispine_pool.py +++ b/tools/build_us_multispine_pool.py @@ -51,6 +51,7 @@ ) from populace.build.us_runtime.h5_io import write_nullable_us_h5 from populace.build.us_runtime.multispine_pool import ( + POOL_HOUSEHOLD_MASS_SHARES, POOL_OPERATOR_ORDER, POOL_RANDOM_SEED, POOL_TIME_PERIOD, @@ -73,7 +74,10 @@ initialize_primary_puf_qrf_chain, run_primary_puf_qrf_chain, ) -from populace.build.us_runtime.puf_support import US_PUF_SUPPORT_FIT_NAME +from populace.build.us_runtime.puf_support import ( + PUF_SUPPORT_MAX_CLONE_SAFE_SOURCE_ID, + US_PUF_SUPPORT_FIT_NAME, +) from populace.frame import Frame __all__ = [ @@ -531,6 +535,17 @@ def _manifest_payload( "build_receipt": loaded.puf_donor_build, }, "assembly_receipt": result.assembly_receipt, + "assembly_contract": { + "household_mass_shares": dict(POOL_HOUSEHOLD_MASS_SHARES), + "clone_safe_source_id_upper_bound": (PUF_SUPPORT_MAX_CLONE_SAFE_SOURCE_ID), + "output_household_weight_kind": result.frame.weights_for( + "household" + ).kind.value, + "output_household_weight_total": result.frame.weights_for( + "household" + ).total, + "mass_log": list(result.frame.mass_log), + }, "provenance_counts": result.provenance_counts, "stage_receipts": result.stage_receipts, "agreement_gate": _agreement_payload(result), From b4ea47854c8fd9e8c276de90f7fb1e9fd62a8dff Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 20:57:33 -0400 Subject: [PATCH 15/94] Harmonize ACS source lineage for assembly --- .../src/populace/build/us_runtime/acs_pums.py | 21 ++-- .../populace-build/tests/test_us_acs_pums.py | 111 ++++++++++++++++-- 2 files changed, 112 insertions(+), 20 deletions(-) diff --git a/packages/populace-build/src/populace/build/us_runtime/acs_pums.py b/packages/populace-build/src/populace/build/us_runtime/acs_pums.py index 0a93f0d3..6e87b8bb 100644 --- a/packages/populace-build/src/populace/build/us_runtime/acs_pums.py +++ b/packages/populace-build/src/populace/build/us_runtime/acs_pums.py @@ -7,6 +7,12 @@ ACS input-mapping stage applies the Census adjustment factors and records which PolicyEngine inputs are native versus transferred. +Raw ``SERIALNO`` remains on the household table. Person lineage uses the +stable, sorted household ID assigned from that key, string-valued ``SPORDER``, +and a sequential row ID. Those generated fields match the ASEC source-frame +lineage dtypes required by pre-operator spine assembly without rewriting a +measured Census value. + Full national archives contain multiple CSV members. Each member is read in bounded chunks, but the final selected-column tables necessarily materialize: the returned :class:`~populace.frame.Frame` itself is the dense base-pool @@ -264,15 +270,12 @@ def build_acs_pums_unit_frame( person["household_id"] = person["household_id"].astype("int64") person = _with_structural_columns(person) person["source_year"] = source.vintage - person["source_household_id"] = person["SERIALNO"].astype(str) - person["source_person_id"] = _required_integer(person, "SPORDER") - person["source_row_id"] = ( - ACS_2024_1YR_SPINE - + ":" - + person["SERIALNO"].astype(str) - + ":" - + person["SPORDER"].astype("int64").astype(str) - ) + person["source_household_id"] = person["household_id"].to_numpy(dtype=np.int64) + person["source_person_id"] = pd.Series( + _required_integer(person, "SPORDER"), + index=person.index, + ).astype(str) + person["source_row_id"] = np.arange(len(person), dtype=np.int64) household_weights = _household_weights(household, person) # SERIALNO belongs on the household table in the returned Frame. Its diff --git a/packages/populace-build/tests/test_us_acs_pums.py b/packages/populace-build/tests/test_us_acs_pums.py index ac1b3b40..52092ee0 100644 --- a/packages/populace-build/tests/test_us_acs_pums.py +++ b/packages/populace-build/tests/test_us_acs_pums.py @@ -3,6 +3,7 @@ from pathlib import Path from zipfile import ZIP_DEFLATED, ZipFile +import numpy as np import pandas as pd import pytest @@ -12,6 +13,8 @@ build_acs_pums_unit_frame, load_acs_pums_tables, ) +from populace.build.us_runtime.spine_assembly import assemble_spines +from populace.frame import US_SCHEMA, Frame, WeightKind, Weights def _write_csv_zip( @@ -122,6 +125,52 @@ def _source(tmp_path: Path) -> AcsPumsSource: return AcsPumsSource(household_zip=household_zip, person_zip=person_zip) +def _asec_shaped_frame() -> Frame: + """Return one ASEC-like row sharing only structural and lineage fields.""" + + person = pd.DataFrame( + { + "person_id": np.asarray([101], dtype=np.int64), + "person_household_id": np.asarray([201], dtype=np.int64), + "person_tax_unit_id": np.asarray([301], dtype=np.int64), + "person_spm_unit_id": np.asarray([401], dtype=np.int64), + "person_family_id": np.asarray([501], dtype=np.int64), + "person_marital_unit_id": np.asarray([601], dtype=np.int64), + "source_year": np.asarray([2024], dtype=np.int64), + "source_household_id": np.asarray([7], dtype=np.int64), + "source_person_id": pd.Series(["7-1"]).astype(str), + "source_row_id": np.asarray([0], dtype=np.int64), + "A_AGE": np.asarray([55], dtype=np.int64), + } + ) + return Frame( + { + "person": person, + "household": pd.DataFrame( + {"household_id": np.asarray([201], dtype=np.int64)} + ), + "tax_unit": pd.DataFrame( + {"tax_unit_id": np.asarray([301], dtype=np.int64)} + ), + "spm_unit": pd.DataFrame( + {"spm_unit_id": np.asarray([401], dtype=np.int64)} + ), + "family": pd.DataFrame({"family_id": np.asarray([501], dtype=np.int64)}), + "marital_unit": pd.DataFrame( + {"marital_unit_id": np.asarray([601], dtype=np.int64)} + ), + }, + US_SCHEMA, + { + "household": Weights( + np.asarray([10.0], dtype=np.float64), + WeightKind.DESIGN, + ) + }, + pd.Series(["asec_2024"], dtype=object), + ) + + def test_load_acs_pums_tables_streams_all_csv_members_and_keeps_native_blanks( tmp_path: Path, ) -> None: @@ -164,12 +213,13 @@ def test_build_acs_pums_unit_frame_preserves_lineage_geography_and_weights( assert household["puma_geoid"].tolist() == ["0612345", "3600100"] assert frame.weights_for("household").values.tolist() == [10.0, 20.0] assert "SERIALNO" not in frame.table("person") - assert frame.table("person")["source_row_id"].tolist() == [ - "acs_2024_1yr:2024HU0000001:1", - "acs_2024_1yr:2024HU0000001:2", - "acs_2024_1yr:2024HU0000001:3", - "acs_2024_1yr:2024HU0000002:1", - ] + person = frame.table("person") + assert person["source_household_id"].tolist() == [1, 1, 1, 2] + assert person["source_person_id"].tolist() == ["1", "2", "3", "1"] + assert person["source_row_id"].tolist() == [0, 1, 2, 3] + assert person["source_household_id"].dtype == np.dtype(np.int64) + assert pd.api.types.is_string_dtype(person["source_person_id"].dtype) + assert person["source_row_id"].dtype == np.dtype(np.int64) assert metadata["weighted_household_population"] == pytest.approx(30.0) @@ -178,11 +228,13 @@ def test_build_acs_pums_unit_frame_derives_only_structural_relationship_fields( ) -> None: pytest.importorskip("microunit") # sanctioned tax-unit constructor (us extra) frame, _metadata = build_acs_pums_unit_frame(_source(tmp_path), chunksize=2) - people = frame.table("person").set_index("source_row_id") - head = people.loc["acs_2024_1yr:2024HU0000001:1"] - spouse = people.loc["acs_2024_1yr:2024HU0000001:2"] - child = people.loc["acs_2024_1yr:2024HU0000001:3"] - lone_head = people.loc["acs_2024_1yr:2024HU0000002:1"] + people = frame.table("person").set_index( + ["source_household_id", "source_person_id"] + ) + head = people.loc[(1, "1")] + spouse = people.loc[(1, "2")] + child = people.loc[(1, "3")] + lone_head = people.loc[(2, "1")] assert (head["A_SPOUSE"], spouse["A_SPOUSE"], child["A_SPOUSE"]) == (2, 1, 0) assert (child["PEPAR1"], child["PEPAR2"]) == (1, 2) @@ -198,6 +250,43 @@ def test_build_acs_pums_unit_frame_derives_only_structural_relationship_fields( assert child["person_marital_unit_id"] != head["person_marital_unit_id"] +def test_built_acs_lineage_assembles_with_asec_without_measured_coercion( + tmp_path: Path, +) -> None: + pytest.importorskip("microunit") # sanctioned tax-unit constructor (us extra) + acs, _metadata = build_acs_pums_unit_frame(_source(tmp_path), chunksize=1) + measured_wages = acs.table("person")["WAGP"].copy() + raw_serials = acs.table("household")["SERIALNO"].copy() + + assembled = assemble_spines( + {"asec": _asec_shaped_frame(), "acs": acs}, + household_mass_shares={"asec": 0.5, "acs": 0.5}, + ) + + person = assembled.table("person") + acs_person = person.loc[person["person_support_channel"].eq("acs")].sort_values( + "source_row_id" + ) + pd.testing.assert_series_equal( + acs_person["WAGP"].reset_index(drop=True), + measured_wages.reset_index(drop=True), + ) + assert person["source_household_id"].dtype == np.dtype(np.int64) + assert ( + person["source_person_id"].dtype + == _asec_shaped_frame().table("person")["source_person_id"].dtype + ) + assert person["source_row_id"].dtype == np.dtype(np.int64) + + household = assembled.table("household") + acs_household = household.loc[household["household_support_channel"].eq("acs")] + pd.testing.assert_series_equal( + acs_household["SERIALNO"].reset_index(drop=True), + raw_serials.reset_index(drop=True), + check_dtype=False, + ) + + def test_load_acs_pums_tables_rejects_duplicate_household_serialno( tmp_path: Path, ) -> None: From 8a5ab94dad8459cb0fcf2ab3376d807503ea34cb Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 21:00:23 -0400 Subject: [PATCH 16/94] Test multispine pool build boundary --- .../tests/test_us_multispine_pool_tool.py | 406 ++++++++++++++++++ 1 file changed, 406 insertions(+) create mode 100644 packages/populace-build/tests/test_us_multispine_pool_tool.py diff --git a/packages/populace-build/tests/test_us_multispine_pool_tool.py b/packages/populace-build/tests/test_us_multispine_pool_tool.py new file mode 100644 index 00000000..fd77736c --- /dev/null +++ b/packages/populace-build/tests/test_us_multispine_pool_tool.py @@ -0,0 +1,406 @@ +"""Small-fixture tests for the terminal US multispine pool build tool.""" + +from __future__ import annotations + +import hashlib +import importlib.util +import json +import sys +from collections.abc import Callable +from pathlib import Path +from types import ModuleType + +import numpy as np +import pandas as pd +import pytest + +from populace.build.gates import GateReport +from populace.build.us_runtime.multispine_pool import PoolStageOutput +from populace.build.us_runtime.puf_support import ( + PUF_SUPPORT_MAX_CLONE_SAFE_SOURCE_ID, +) +from populace.build.us_runtime.support_provenance import ( + support_channel_column, + support_clone_index_column, +) +from populace.frame import US_SCHEMA, Frame, WeightKind, Weights + + +@pytest.fixture(scope="module") +def pool_tool() -> ModuleType: + root = Path(__file__).resolve().parents[3] + path = root / "tools" / "build_us_multispine_pool.py" + spec = importlib.util.spec_from_file_location( + "build_us_multispine_pool_fixture", + path, + ) + module = importlib.util.module_from_spec(spec) + assert spec.loader is not None + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +def _source_frame(*, measured_offset: float = 0.0) -> Frame: + ids = np.asarray([1, 2], dtype=np.int64) + person = pd.DataFrame( + { + "person_id": ids, + "person_household_id": ids, + "person_tax_unit_id": ids, + "person_spm_unit_id": ids, + "person_family_id": ids, + "person_marital_unit_id": ids, + "age": np.asarray([30.0, 50.0]), + "measured": np.asarray([1.0, 2.0]) + measured_offset, + } + ) + tables = { + "person": person, + **{ + entity: pd.DataFrame({f"{entity}_id": ids}) + for entity in US_SCHEMA.group_entities + }, + } + return Frame( + tables, + US_SCHEMA, + { + "household": Weights( + np.asarray([2.0, 2.0]), + WeightKind.DESIGN, + ) + }, + pd.Series(["fixture", "fixture"], dtype=object), + ) + + +def _replace_person( + frame: Frame, + person: pd.DataFrame, + *, + preserve_metadata: bool = True, +) -> Frame: + tables = {entity: frame.table(entity).copy() for entity in frame.entities} + tables["person"] = person + return Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata if preserve_metadata else None, + ) + + +def _red_pool_result(pool_tool: ModuleType, tmp_path: Path): + order: list[str] = [] + + def stage( + name: str, + transform: Callable[[pd.DataFrame], None], + ) -> Callable[[Frame], PoolStageOutput]: + def apply(frame: Frame) -> PoolStageOutput: + order.append(name) + person = frame.table("person").copy() + assert set( + person[support_clone_index_column("person")].astype(int) + ) == {0, 1} + assert set(person[support_channel_column("person")]) == { + "asec", + "acs", + } + transform(person) + return PoolStageOutput( + _replace_person(frame, person), + {"fixture_stage": name}, + ) + + return apply + + def transfer(person: pd.DataFrame) -> None: + person["fixture_transfer"] = person["measured"] + + def derive(person: pd.DataFrame) -> None: + person["fixture_derived"] = person["fixture_transfer"] + 1.0 + + def seed(person: pd.DataFrame) -> None: + person["fixture_seed"] = person["fixture_derived"] > 0.0 + + def simulate(person: pd.DataFrame) -> None: + channels = person[support_channel_column("person")] + person["ssi"] = np.where(channels.eq("asec"), 1.0, 100.0) + + result = pool_tool.build_multispine_pool( + _source_frame(), + _source_frame(measured_offset=99.0), + puf_donor=pd.DataFrame(), + primary_qrf_checkpoint_dir=tmp_path / "unused-qrf", + impute=stage("impute", transfer), + derive=stage("derive", derive), + seed=stage("seed", seed), + simulate=stage("simulate", simulate), + ) + + assert order == ["impute", "derive", "seed", "simulate"] + assert not result.agreement_gate.passed + assert not result.simulation_ready + assert result.agreement_gate.name == "us_spine_agreement" + assert result.agreement_gate.details["tolerances"] == { + "incidence_ratio_bounds": [0.8, 1.25], + "max_quantile_envelope_distance": 0.25, + } + assert "ssi" not in result.frame.table("person") + return result + + +def test_parser_exposes_only_five_pinned_inputs_and_out( + pool_tool: ModuleType, +) -> None: + parser = pool_tool._parser() + actions = { + action.dest: action + for action in parser._actions + if action.dest != "help" + } + pairs = ( + ("asec_pre_clone_h5", "asec_pre_clone_h5_sha256"), + ("acs_household_zip", "acs_household_zip_sha256"), + ("acs_person_zip", "acs_person_zip_sha256"), + ("puf_h5", "puf_h5_sha256"), + ("puf_source_year_csv", "puf_source_year_csv_sha256"), + ) + expected_destinations = { + destination for pair in pairs for destination in pair + } | {"out"} + + assert set(actions) == expected_destinations + assert all(action.required for action in actions.values()) + assert actions["out"].option_strings == ["--out"] + for path_destination, sha_destination in pairs: + assert actions[path_destination].type is Path + assert actions[sha_destination].type is pool_tool._sha256_argument + assert len(actions[path_destination].option_strings) == 1 + assert len(actions[sha_destination].option_strings) == 1 + + option_names = { + option + for action in actions.values() + for option in action.option_strings + } + assert not any( + forbidden in option + for option in option_names + for forbidden in ("tolerance", "target", "per-target") + ) + + +def test_sha_mismatch_refuses_before_loading_or_writing( + pool_tool: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + source_paths = { + "asec-pre-clone-h5": tmp_path / "asec.h5", + "acs-household-zip": tmp_path / "household.zip", + "acs-person-zip": tmp_path / "person.zip", + "puf-h5": tmp_path / "puf.h5", + "puf-source-year-csv": tmp_path / "puf.csv", + } + for path in source_paths.values(): + path.write_bytes(b"fixture input") + + called = {"load": False, "write": False} + + def unexpected_load(*_args, **_kwargs): + called["load"] = True + raise AssertionError("SHA mismatch must precede source-frame loading.") + + def unexpected_write(*_args, **_kwargs): + called["write"] = True + raise AssertionError("SHA mismatch must precede output writing.") + + monkeypatch.setattr(pool_tool, "_load_inputs", unexpected_load) + monkeypatch.setattr(pool_tool, "_write_outputs", unexpected_write) + output = tmp_path / "pool.h5" + argv: list[str] = [] + for option, path in source_paths.items(): + argv.extend([f"--{option}", str(path)]) + argv.extend([f"--{option}-sha256", "0" * 64]) + argv.extend(["--out", str(output)]) + + with pytest.raises(ValueError, match="ASEC pre-clone.*SHA-256 mismatch"): + pool_tool.main(argv) + + assert called == {"load": False, "write": False} + assert not output.exists() + assert not output.with_suffix(".manifest.json").exists() + assert not output.with_suffix(".agreement.json").exists() + + +def test_synthetic_two_spine_path_reaches_fixed_red_terminal_gate( + pool_tool: ModuleType, + tmp_path: Path, +) -> None: + result = _red_pool_result(pool_tool, tmp_path) + + assert result.assembly_receipt["channels"] == ["asec", "acs"] + assert result.assembly_receipt["native_row_counts"]["person"] == { + "asec": 2, + "acs": 2, + } + assert result.provenance_counts["person"] == { + "rows": 8, + "by_source_channel": {"asec": 4, "acs": 4}, + "by_clone_index": {"0": 4, "1": 4}, + "by_source_channel_and_clone_index": { + "asec": {"0": 2, "1": 2}, + "acs": {"0": 2, "1": 2}, + }, + } + assert result.stage_receipts == { + stage: {"fixture_stage": stage} + for stage in ("impute", "derive", "seed", "simulate") + } + assert any( + "person/simulated_output/ssi/acs_vs_asec" in failure + for failure in result.agreement_gate.failures + ) + + +def test_red_outputs_preserve_receipts_and_exclude_simulation_output( + pool_tool: ModuleType, + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + result = _red_pool_result(pool_tool, tmp_path) + outputs = pool_tool._output_paths(tmp_path / "pool.h5") + source_manifest = pool_tool.load_acs_source_manifest() + + verified_inputs = {} + for index, role in enumerate( + ( + "asec_pre_clone", + "acs_household", + "acs_person", + "processed_puf", + "puf_source_year", + ), + start=1, + ): + path = tmp_path / f"{role}.fixture" + path.write_bytes(f"input-{index}".encode()) + digest = hashlib.sha256(path.read_bytes()).hexdigest() + verified_inputs[role] = pool_tool._VerifiedInput( + role=role, + path=path, + expected_sha256=digest, + actual_sha256=digest, + size_bytes=path.stat().st_size, + ) + loaded = pool_tool._LoadedInputs( + asec=_source_frame(), + acs=_source_frame(measured_offset=99.0), + puf_donor=pd.DataFrame({"RECID": [1]}), + asec_checkpoint={"artifact": "fixture-pre-clone"}, + acs_build={"artifact": "fixture-unit-frame"}, + acs_native_inputs={"person": {"age": {"source": "fixture"}}}, + puf_donor_build={"artifact": "fixture-donor"}, + ) + + pool_tool._write_outputs( + result, + outputs=outputs, + verified_inputs=verified_inputs, + acs_source_manifest=source_manifest, + loaded=loaded, + ) + + manifest = json.loads(outputs.manifest.read_text(encoding="utf-8")) + diagnostics = json.loads( + outputs.agreement_diagnostics.read_text(encoding="utf-8") + ) + expected_gate = GateReport((result.agreement_gate,)).to_manifest() + + assert manifest["status"] == "agreement_failed" + assert manifest["simulation_ready"] is False + assert manifest["calibration_applied"] is False + assert manifest["calibration"]["applied"] is False + assert manifest["assembly_receipt"] == result.assembly_receipt + assert manifest["provenance_counts"] == result.provenance_counts + assert manifest["stage_receipts"] == result.stage_receipts + assert manifest["agreement_gate"] == expected_gate + assert diagnostics["agreement_gate"] == expected_gate + assert diagnostics["simulation_ready"] is False + assert manifest["provenance_pins"] == { + role: pin.to_manifest() for role, pin in verified_inputs.items() + } + assert manifest["pool_h5"]["formula_outputs_persisted"] is False + assert manifest["pool_h5"]["input_only"] is True + + with pd.HDFStore(outputs.pool_h5, mode="r") as store: + assert "ssi" not in store["person"].columns + + +def test_clone_safe_id_error_surfaces_unchanged_through_tool( + pool_tool: ModuleType, + tmp_path: Path, +) -> None: + asec = _source_frame() + tables = {entity: asec.table(entity).copy() for entity in asec.entities} + tables["person"].loc[0, "person_id"] = ( + PUF_SUPPORT_MAX_CLONE_SAFE_SOURCE_ID + 1 + ) + invalid = Frame( + tables, + asec.schema, + {"household": asec.weights_for("household")}, + asec.strata, + ) + + def unreachable(_frame: Frame) -> PoolStageOutput: + raise AssertionError("Assembly errors must precede every pool stage.") + + with pytest.raises(ValueError, match="Spine 'asec'.*clone-safe bound"): + pool_tool.build_multispine_pool( + invalid, + _source_frame(), + puf_donor=pd.DataFrame(), + primary_qrf_checkpoint_dir=tmp_path / "unused-qrf", + impute=unreachable, + derive=unreachable, + seed=unreachable, + simulate=unreachable, + ) + + +def test_assembly_receipt_loss_surfaces_unchanged_through_tool( + pool_tool: ModuleType, + tmp_path: Path, +) -> None: + def no_op(frame: Frame) -> PoolStageOutput: + return PoolStageOutput(frame) + + def drop_receipt(frame: Frame) -> PoolStageOutput: + return PoolStageOutput( + _replace_person( + frame, + frame.table("person").copy(), + preserve_metadata=False, + ) + ) + + with pytest.raises( + ValueError, + match="multispine pool derive output:.*no assembly manifest", + ): + pool_tool.build_multispine_pool( + _source_frame(), + _source_frame(), + puf_donor=pd.DataFrame(), + primary_qrf_checkpoint_dir=tmp_path / "unused-qrf", + impute=no_op, + derive=drop_receipt, + seed=no_op, + simulate=no_op, + ) From f9a3e1b50fe605796e4239f104cbb2951e302997 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 21:01:46 -0400 Subject: [PATCH 17/94] Bind resumable QRF to verified pool inputs --- PROGRESS.md | 19 +++-- .../tests/test_us_multispine_pool_tool.py | 62 +++++++++++----- tools/build_us_multispine_pool.py | 74 +++++++++++++++++-- 3 files changed, 124 insertions(+), 31 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index e767de52..07c26360 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -73,12 +73,21 @@ - Extended the AST guard transitively from the new CLI through its US runtime imports, with no new source-provenance owner exceptions, and rejected the retired late-assembly graph. +- Bound resumable primary-QRF checkpoints to the exact five verified input + digests and sizes. A missing or changed binding now refuses stale + predictions instead of allowing a manifest to claim different input bytes. +- Harmonized only the ACS adapter's generated lineage fields to ASEC-compatible + dtypes before assembly. Raw `SERIALNO`, wages, nulls, and other measured ACS + values remain unchanged; a real tiny ASEC/ACS adapter comparison found no + remaining shared-column dtype mismatch. +- Added focused tool-boundary fixtures covering the exact CLI surface, + refuse-before-load SHA checks, the complete ordered two-spine seam and red + terminal gate, failure H5/diagnostic/manifest receipts, unchanged #581 + clone-bound and receipt-loss errors, and stale QRF checkpoint refusal. ## Next -1. Add synthetic full-path CLI tests for red agreement receipts, manifest - contents, structural #581 errors, and pre-load SHA refusal. -2. Verify and fix the ACS/ASEC source-lineage dtype boundary discovered during - the production input audit. -3. Run focused verification, update the committed ledger, and write +1. Add one wired small-fixture test using the real raw-preserving ACS transfer + entrypoint between clone and terminal gate. +2. Run focused verification, update this committed ledger, and write the external review worklog. diff --git a/packages/populace-build/tests/test_us_multispine_pool_tool.py b/packages/populace-build/tests/test_us_multispine_pool_tool.py index fd77736c..3b51d1cc 100644 --- a/packages/populace-build/tests/test_us_multispine_pool_tool.py +++ b/packages/populace-build/tests/test_us_multispine_pool_tool.py @@ -103,9 +103,10 @@ def stage( def apply(frame: Frame) -> PoolStageOutput: order.append(name) person = frame.table("person").copy() - assert set( - person[support_clone_index_column("person")].astype(int) - ) == {0, 1} + assert set(person[support_clone_index_column("person")].astype(int)) == { + 0, + 1, + } assert set(person[support_channel_column("person")]) == { "asec", "acs", @@ -159,9 +160,7 @@ def test_parser_exposes_only_five_pinned_inputs_and_out( ) -> None: parser = pool_tool._parser() actions = { - action.dest: action - for action in parser._actions - if action.dest != "help" + action.dest: action for action in parser._actions if action.dest != "help" } pairs = ( ("asec_pre_clone_h5", "asec_pre_clone_h5_sha256"), @@ -170,9 +169,9 @@ def test_parser_exposes_only_five_pinned_inputs_and_out( ("puf_h5", "puf_h5_sha256"), ("puf_source_year_csv", "puf_source_year_csv_sha256"), ) - expected_destinations = { - destination for pair in pairs for destination in pair - } | {"out"} + expected_destinations = {destination for pair in pairs for destination in pair} | { + "out" + } assert set(actions) == expected_destinations assert all(action.required for action in actions.values()) @@ -184,9 +183,7 @@ def test_parser_exposes_only_five_pinned_inputs_and_out( assert len(actions[sha_destination].option_strings) == 1 option_names = { - option - for action in actions.values() - for option in action.option_strings + option for action in actions.values() for option in action.option_strings } assert not any( forbidden in option @@ -238,6 +235,39 @@ def unexpected_write(*_args, **_kwargs): assert not output.with_suffix(".agreement.json").exists() +def test_primary_qrf_resume_refuses_a_changed_input_binding( + pool_tool: ModuleType, + tmp_path: Path, +) -> None: + checkpoint_dir = tmp_path / "primary-qrf" + checkpoint_dir.mkdir() + (checkpoint_dir / pool_tool.PRIMARY_QRF_MANIFEST_FILENAME).write_text( + "{}\n", + encoding="utf-8", + ) + original = { + "artifact_kind": "fixture_binding", + "schema_version": 1, + "inputs": {"processed_puf": {"sha256": "a" * 64}}, + } + pool_tool._atomic_write_json( + checkpoint_dir / pool_tool._PRIMARY_QRF_INPUT_BINDING_FILENAME, + original, + ) + changed = { + **original, + "inputs": {"processed_puf": {"sha256": "b" * 64}}, + } + + with pytest.raises(ValueError, match="refusing to reuse stale predictions"): + pool_tool._initialize_or_resume_primary_qrf( + _source_frame(), + pd.DataFrame(), + checkpoint_dir, + input_binding=changed, + ) + + def test_synthetic_two_spine_path_reaches_fixed_red_terminal_gate( pool_tool: ModuleType, tmp_path: Path, @@ -317,9 +347,7 @@ def test_red_outputs_preserve_receipts_and_exclude_simulation_output( ) manifest = json.loads(outputs.manifest.read_text(encoding="utf-8")) - diagnostics = json.loads( - outputs.agreement_diagnostics.read_text(encoding="utf-8") - ) + diagnostics = json.loads(outputs.agreement_diagnostics.read_text(encoding="utf-8")) expected_gate = GateReport((result.agreement_gate,)).to_manifest() assert manifest["status"] == "agreement_failed" @@ -348,9 +376,7 @@ def test_clone_safe_id_error_surfaces_unchanged_through_tool( ) -> None: asec = _source_frame() tables = {entity: asec.table(entity).copy() for entity in asec.entities} - tables["person"].loc[0, "person_id"] = ( - PUF_SUPPORT_MAX_CLONE_SAFE_SOURCE_ID + 1 - ) + tables["person"].loc[0, "person_id"] = PUF_SUPPORT_MAX_CLONE_SAFE_SOURCE_ID + 1 invalid = Frame( tables, asec.schema, diff --git a/tools/build_us_multispine_pool.py b/tools/build_us_multispine_pool.py index 220f3012..7b061c76 100644 --- a/tools/build_us_multispine_pool.py +++ b/tools/build_us_multispine_pool.py @@ -96,6 +96,8 @@ _PRIMARY_QRF_N_ESTIMATORS = 100 _ACS_TRANSFER_N_ESTIMATORS = 100 +_PRIMARY_QRF_INPUT_BINDING_FILENAME = "pool-input-binding.json" +_PRIMARY_QRF_INPUT_BINDING_SCHEMA_VERSION = 1 _LOWERCASE_SHA256 = re.compile(r"[0-9a-f]{64}") type PoolOperator = Callable[[Frame], PoolStageOutput] @@ -357,13 +359,51 @@ def _primary_qrf_manifest_path(checkpoint_dir: Path) -> Path: return checkpoint_dir / PRIMARY_QRF_MANIFEST_FILENAME +def _primary_qrf_input_binding_path(checkpoint_dir: Path) -> Path: + return checkpoint_dir / _PRIMARY_QRF_INPUT_BINDING_FILENAME + + +def _checkpoint_input_binding( + verified_inputs: Mapping[str, _VerifiedInput], +) -> dict[str, object]: + return { + "artifact_kind": "populace_us_multispine_primary_qrf_input_binding", + "schema_version": _PRIMARY_QRF_INPUT_BINDING_SCHEMA_VERSION, + "period": POOL_TIME_PERIOD, + "seed": POOL_RANDOM_SEED, + "n_estimators": _PRIMARY_QRF_N_ESTIMATORS, + "inputs": { + role: { + "sha256": pin.actual_sha256, + "size_bytes": pin.size_bytes, + } + for role, pin in verified_inputs.items() + }, + } + + def _initialize_or_resume_primary_qrf( frame: Frame, donor: pd.DataFrame, checkpoint_dir: Path, + *, + input_binding: Mapping[str, object], ) -> None: manifest_path = _primary_qrf_manifest_path(checkpoint_dir) + binding_path = _primary_qrf_input_binding_path(checkpoint_dir) + expected_binding = _json_ready(input_binding) if manifest_path.is_file(): + if not binding_path.is_file(): + raise ValueError( + "Primary QRF checkpoint has no pool-input provenance binding: " + f"{binding_path}." + ) + observed_binding = _read_json_object(binding_path) + if observed_binding != expected_binding: + raise ValueError( + "Primary QRF checkpoint input binding differs from the verified " + "pool inputs; refusing to reuse stale predictions." + ) return if checkpoint_dir.exists(): if not checkpoint_dir.is_dir(): @@ -383,6 +423,7 @@ def _initialize_or_resume_primary_qrf( seed=POOL_RANDOM_SEED, n_estimators=_PRIMARY_QRF_N_ESTIMATORS, ) + _atomic_write_json(binding_path, input_binding) def _impute_pool( @@ -390,8 +431,14 @@ def _impute_pool( *, puf_donor: pd.DataFrame, checkpoint_dir: Path, + checkpoint_input_binding: Mapping[str, object], ) -> PoolStageOutput: - _initialize_or_resume_primary_qrf(frame, puf_donor, checkpoint_dir) + _initialize_or_resume_primary_qrf( + frame, + puf_donor, + checkpoint_dir, + input_binding=checkpoint_input_binding, + ) run_primary_puf_qrf_chain(checkpoint_dir) tail_bound_diagnostics: list[dict[str, object]] = [] @@ -437,12 +484,15 @@ def _impute_pool( ) qrf_manifest_path = _primary_qrf_manifest_path(checkpoint_dir) + qrf_binding_path = _primary_qrf_input_binding_path(checkpoint_dir) return PoolStageOutput( transferred.frame, { "primary_puf_qrf": { "checkpoint_manifest": _read_json_object(qrf_manifest_path), "checkpoint_manifest_sha256": _file_sha256(qrf_manifest_path), + "input_binding": _read_json_object(qrf_binding_path), + "input_binding_sha256": _file_sha256(qrf_binding_path), "n_estimators": _PRIMARY_QRF_N_ESTIMATORS, "tail_bound_diagnostics": tail_bound_diagnostics, }, @@ -467,6 +517,7 @@ def build_multispine_pool( *, puf_donor: pd.DataFrame, primary_qrf_checkpoint_dir: Path, + checkpoint_input_binding: Mapping[str, object] | None = None, impute: PoolOperator | None = None, derive: PoolOperator = derive_multispine_pool_inputs, seed: PoolOperator = seed_multispine_pool_inputs, @@ -479,17 +530,23 @@ def build_multispine_pool( its registry or fixed tolerances. """ - impute_operator = ( - ( - lambda frame: _impute_pool( + if impute is None: + if checkpoint_input_binding is None: + raise ValueError( + "Production pool imputation requires a verified checkpoint " + "input binding." + ) + + def impute_operator(frame: Frame) -> PoolStageOutput: + return _impute_pool( frame, puf_donor=puf_donor, checkpoint_dir=primary_qrf_checkpoint_dir, + checkpoint_input_binding=checkpoint_input_binding, ) - ) - if impute is None - else impute - ) + + else: + impute_operator = impute return run_multispine_pool_path( asec, acs, @@ -677,6 +734,7 @@ def main(argv: list[str] | None = None) -> int: loaded.acs, puf_donor=loaded.puf_donor, primary_qrf_checkpoint_dir=outputs.primary_qrf_checkpoint_dir, + checkpoint_input_binding=_checkpoint_input_binding(verified_inputs), ) _write_outputs( result, From 136ea1ede8e3e4906df89c9123c9cfe03f575648 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 21:03:35 -0400 Subject: [PATCH 18/94] Exercise real transfer through pool gate --- PROGRESS.md | 8 +- .../tests/test_us_multispine_pool_tool.py | 115 ++++++++++++++++++ 2 files changed, 120 insertions(+), 3 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 07c26360..4a85d579 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -84,10 +84,12 @@ refuse-before-load SHA checks, the complete ordered two-spine seam and red terminal gate, failure H5/diagnostic/manifest receipts, unchanged #581 clone-bound and receipt-loss errors, and stale QRF checkpoint refusal. +- Added a wired two-spine fixture that invokes the real missing-cell ACS + transfer entrypoint after assembly and cloning, proves existing values remain + untouched while missing peer-spine cells are imputed, and then reaches the + unchanged terminal agreement gate. ## Next -1. Add one wired small-fixture test using the real raw-preserving ACS transfer - entrypoint between clone and terminal gate. -2. Run focused verification, update this committed ledger, and write +1. Run focused verification, update this committed ledger, and write the external review worklog. diff --git a/packages/populace-build/tests/test_us_multispine_pool_tool.py b/packages/populace-build/tests/test_us_multispine_pool_tool.py index 3b51d1cc..06fc973a 100644 --- a/packages/populace-build/tests/test_us_multispine_pool_tool.py +++ b/packages/populace-build/tests/test_us_multispine_pool_tool.py @@ -14,7 +14,9 @@ import pandas as pd import pytest +import populace.build.us_runtime.acs_transfer as acs_transfer_module from populace.build.gates import GateReport +from populace.build.us_runtime.acs_transfer import transfer_acs_inputs from populace.build.us_runtime.multispine_pool import PoolStageOutput from populace.build.us_runtime.puf_support import ( PUF_SUPPORT_MAX_CLONE_SAFE_SOURCE_ID, @@ -93,6 +95,58 @@ def _replace_person( ) +class _MeanQRF: + def __init__(self, *, n_estimators: int, seed: int) -> None: + self.n_estimators = n_estimators + self.seed = seed + + def fit( + self, + frame: Frame, + predictors: list[str], + targets: list[str], + *, + weights: str, + ) -> _MeanFitted: + assert ( + weights == frame.resolve_weights(frame.column_entity(targets[0])).kind.value + ) + table = frame.table(frame.column_entity(targets[0])) + return _MeanFitted( + {target: float(table[target].mean()) for target in targets}, + weights, + ) + + +class _MeanFitted: + def __init__(self, means: dict[str, float], weight_kind: str) -> None: + self.means = means + self.weight_kind = weight_kind + + def predict(self, frame: pd.DataFrame) -> pd.DataFrame: + return pd.DataFrame( + { + target: np.full(len(frame), mean, dtype=np.float64) + for target, mean in self.means.items() + }, + index=frame.index, + ) + + +def _transfer_source_frame(targets: list[float]) -> Frame: + frame = _source_frame() + tables = {entity: frame.table(entity).copy() for entity in frame.entities} + tables["person"]["is_female"] = np.asarray([False, True], dtype=bool) + tables["person"]["fixture_transfer"] = np.asarray(targets, dtype=np.float64) + tables["household"]["state_fips"] = np.asarray([6, 36], dtype=np.int64) + return Frame( + tables, + frame.schema, + {"household": frame.weights_for("household")}, + frame.strata, + ) + + def _red_pool_result(pool_tool: ModuleType, tmp_path: Path): order: list[str] = [] @@ -298,6 +352,67 @@ def test_synthetic_two_spine_path_reaches_fixed_red_terminal_gate( ) +def test_wired_path_uses_real_raw_preserving_transfer_before_gate( + pool_tool: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + transfer_receipts = [] + + def impute(frame: Frame) -> PoolStageOutput: + transferred = transfer_acs_inputs( + frame, + frame, + target_families={"person": {"fixture": ("fixture_transfer",)}}, + seed=0, + n_estimators=3, + ) + transfer_receipts.extend(transferred.imputed_inputs) + return PoolStageOutput( + transferred.frame, + {"fit_records": list(transferred.fit_records)}, + ) + + def no_op(frame: Frame) -> PoolStageOutput: + return PoolStageOutput(frame) + + def simulate(frame: Frame) -> PoolStageOutput: + person = frame.table("person").copy() + person["ssi"] = person["fixture_transfer"] + return PoolStageOutput(_replace_person(frame, person)) + + result = pool_tool.build_multispine_pool( + _transfer_source_frame([10.0, 20.0]), + _transfer_source_frame([np.nan, np.nan]), + puf_donor=pd.DataFrame(), + primary_qrf_checkpoint_dir=tmp_path / "unused-qrf", + impute=impute, + derive=no_op, + seed=no_op, + simulate=simulate, + ) + + person = result.frame.table("person") + channels = person[support_channel_column("person")] + assert sorted(person.loc[channels.eq("asec"), "fixture_transfer"]) == [ + 10.0, + 10.0, + 20.0, + 20.0, + ] + assert person.loc[channels.eq("acs"), "fixture_transfer"].tolist() == [ + 15.0, + 15.0, + 15.0, + 15.0, + ] + assert sum(item.imputed_recipient_rows for item in transfer_receipts) == 4 + assert not result.agreement_gate.passed + assert result.agreement_gate.name == "us_spine_agreement" + assert "ssi" not in person + + def test_red_outputs_preserve_receipts_and_exclude_simulation_output( pool_tool: ModuleType, tmp_path: Path, From f3cc1dca713e4f2df7c051e3b0fc23fb6d1b3858 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 21:09:23 -0400 Subject: [PATCH 19/94] Record increment 2 validation handoff --- PROGRESS.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 4a85d579..78fa7327 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -6,6 +6,7 @@ - Worktree: `/Users/maxghenis/PolicyEngine/_worktrees/populace-578-inc2` - Scope: code and small synthetic-fixture tests only; no dataset downloads or full-data builds. +- Implementation status: complete and locally committed; no push performed. - Remote `main` was verified through GitHub at merge commit `6c14a0a8590402d1805a24e55ca5f017f39dc281`. - The sandbox blocked `git fetch` at DNS resolution. The local worktree is @@ -13,6 +14,8 @@ `956dc0a3dd5ceaffae0e2007d98dc951e773d389`, whose tree is the merge result. Rebase onto fetched `origin/main` remains a handoff prerequisite if network access is not restored in this lane. +- A final fetch retry after implementation failed with the same + `Could not resolve host: github.com` error. ## Done @@ -88,8 +91,17 @@ transfer entrypoint after assembly and cloning, proves existing values remain untouched while missing peer-spine cells are imputed, and then reaches the unchanged terminal agreement gate. +- Ruff passed across all 19 changed Python files, all changed files passed + formatting and whitespace checks, and 250 focused small-fixture tests passed + across the new pool path plus adjacent assembly, agreement, ACS, PUF QRF/tail, + take-up, QBI, H5-shim, and checkpoint contracts. +- Confirmed the worktree is clean and every coherent implementation step is a + local commit. ## Next -1. Run focused verification, update this committed ledger, and write - the external review worklog. +1. From a network-enabled main session, fetch `origin/main` and rebase this + branch onto a tip at or past `6c14a0a`, then rerun the focused checks. +2. Push/open the PR from the main session. +3. Run the separately sized full-data build and calibration/k-ladder lane; + neither was executed here. From c4c9734c585f44bc232ebfec4aaae5d3c3ebcab8 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 21:12:23 -0400 Subject: [PATCH 20/94] Keep PROGRESS.md at origin/main (root journals stay out of PRs) Co-Authored-By: Claude Fable 5 --- PROGRESS.md | 137 +++++++++++++++------------------------------------- 1 file changed, 39 insertions(+), 98 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 78fa7327..f3158fdd 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,107 +1,48 @@ -# Progress: populace#578 increment 2 +# Progress ## State -- Branch: `multispine-pool-build-578` -- Worktree: `/Users/maxghenis/PolicyEngine/_worktrees/populace-578-inc2` -- Scope: code and small synthetic-fixture tests only; no dataset downloads or - full-data builds. -- Implementation status: complete and locally committed; no push performed. -- Remote `main` was verified through GitHub at merge commit - `6c14a0a8590402d1805a24e55ca5f017f39dc281`. -- The sandbox blocked `git fetch` at DNS resolution. The local worktree is - temporarily rooted at merged PR #581's exact head - `956dc0a3dd5ceaffae0e2007d98dc951e773d389`, whose tree is the merge result. - Rebase onto fetched `origin/main` remains a handoff prerequisite if network - access is not restored in this lane. -- A final fetch retry after implementation failed with the same - `Could not resolve host: github.com` error. +Populace #516 whole-row donor outlier screen is complete on +`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515 +interim carve merged as #525). The `puf_tax_detail` donor now drops tax units +whose grouped raw mortgage interest reaches $10M before the #515 carve +(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T +of phantom mortgage-interest mass), with the checkpoint schema bumped to v3 +so post-carve pre-screen checkpoints rebuild. ## Done -- Read populace#578, its governing “Scope hardening” section, and the UK parity - audit comment. -- Read merged populace#581's contract, review note, changed-file inventory, and - merge metadata. -- Verified GitHub's current `main` tip is `6c14a0a`. -- Created the requested branch and worktree without modifying the existing - checkout. -- Audited the legacy ACS multispine tool's consumers. The local-release builder - imports its H5 helpers and the published build recipe names its CLI, so the - legacy path will remain only as a deprecated compatibility shim. -- Traced the production stage contracts from the ASEC pre-clone checkpoint and - byte-pinned ACS PUMS acquisition through assembly, PUF cloning and transfer, - derivation, seeding, SSI simulation, and the fixed agreement gate. -- Identified two correctness fixes required by the new ordering: - `transfer_acs_inputs` must fill missing cells without overwriting measured - values, and every post-assembly `Frame` reconstruction must preserve the - #581 metadata receipt. -- Ran the focused #581 assembly, agreement, clone-routing, and AST-guard tests - against the starting tree: 44 passed. -- Added the canonical order-bearing runtime seam: - assemble → clone → impute → derive → seed → simulate → terminal agreement. - The simulated formula-output view is separate from the returned input-only - pool, and every operator boundary revalidates the immutable #581 receipt. -- Added provenance-owner reporting helpers for JSON-ready assembly receipts and - per-entity source-channel/clone-index counts; population operators still - receive no source-routing interface. -- Added small two-source tests for the ordered path, batched red agreement, - receipt loss, clone-safe ID refusal, manifest counts, and default take-up - inventory coverage. The new seam and the full spine-blindness guard pass. -- Added a public, bounded PUF donor loader that retains the existing - source-year E00100 alignment and processed-H5 refusal contracts. -- Made the TANF/EITC seed stage pool-safe: clone-stable IDs are a rowwise - fallback for draw keys, and assembled-pool cells already carrying values are - never overwritten. Other unresolved take-up defaults are filled only with - the live engine default and explicitly labeled as such; a missing - transfer-owned flag still fails closed. -- Added fixed-batch SSI materialization on an ephemeral, receipt-preserving - gate view. Formula-owned `ssi` is deliberately absent from the returned - input pool. -- Added a disposable simulation projection that fills any still-null engine - inputs from the live engine defaults solely while materializing SSI. The - nullable pool remains untouched, and every temporary fill is receipted. -- Added the bounded ASEC pre-clone checkpoint loader: it requires the exact - `pre_clone_enrichment` stage binding, revalidates the stored frame identity, - and rejects non-US or invalid household-weight artifacts. -- Added an explicit spine-blind derivation stage that completes the Schedule D - memo input at tax-unit grain without rewriting existing values, then applies - the shared QBI identity reconciliation while preserving the assembly receipt. -- Added `tools/build_us_multispine_pool.py`: its CLI requires five explicit - path/SHA pairs plus `--out`, rechecks packaged ACS byte pins without - downloading, runs the fixed pool order, retains the resumable fixed-parameter - QRF checkpoint, and writes an input-only nullable H5, terminal agreement - diagnostics, and an authoritative manifest. Red agreement returns nonzero - with `simulation_ready=false`; calibration remains downstream. -- Extended the AST guard transitively from the new CLI through its US runtime - imports, with no new source-provenance owner exceptions, and rejected the - retired late-assembly graph. -- Bound resumable primary-QRF checkpoints to the exact five verified input - digests and sizes. A missing or changed binding now refuses stale - predictions instead of allowing a manifest to claim different input bytes. -- Harmonized only the ACS adapter's generated lineage fields to ASEC-compatible - dtypes before assembly. Raw `SERIALNO`, wages, nulls, and other measured ACS - values remain unchanged; a real tiny ASEC/ACS adapter comparison found no - remaining shared-column dtype mismatch. -- Added focused tool-boundary fixtures covering the exact CLI surface, - refuse-before-load SHA checks, the complete ordered two-spine seam and red - terminal gate, failure H5/diagnostic/manifest receipts, unchanged #581 - clone-bound and receipt-loss errors, and stale QRF checkpoint refusal. -- Added a wired two-spine fixture that invokes the real missing-cell ACS - transfer entrypoint after assembly and cloning, proves existing values remain - untouched while missing peer-spine cells are imputed, and then reaches the - unchanged terminal agreement gate. -- Ruff passed across all 19 changed Python files, all changed files passed - formatting and whitespace checks, and 250 focused small-fixture tests passed - across the new pool path plus adjacent assembly, agreement, ACS, PUF QRF/tail, - take-up, QBI, H5-shim, and checkpoint contracts. -- Confirmed the worktree is clean and every coherent implementation step is a - local commit. +- Confirmed a clean starting worktree at `aef1c56`. +- Read the repository guidance and established the #515 donor carve as the + screen's required downstream boundary. +- Started source-level audits of every donor-frame consumer, checkpoint + validation, row-count pins, and existing donor-fact summaries. +- Attempted the requested GitNexus impact workflow; the managed filesystem + denied its global registry write. Its local index also exposed a broad + `build/` ignore mismatch, so the completed impact audit uses direct source + call sites and tests. +- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the + structural rationale and pinned-artifact receipts. +- Added a whole-row screen on grouped raw person `home_mortgage_interest` + after tax-unit assembly, before the #515 carve, with retained-index reset. +- Confirmed no downstream consumer pairs donor rows to the original HDF arrays + or carries a stale donor-length vector; values and weights always originate + from the same screened frame. +- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale + checkpoint regression track the live constant while retaining literal-v1 + corruptions. +- Added regression coverage for the exact grouped boundary, whole-row removal, + retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant. +- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets + 139; populace-data 138 with 1 skip. The directly affected tail-bound suite + adds 12 passes. Ruff format/check and `git diff --check` are clean. +- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line + audit, expected 208,611-row real-artifact effect, verification results, count + sweep, and deliberately untouched surfaces. ## Next -1. From a network-enabled main session, fetch `origin/main` and rebase this - branch onto a tip at or past `6c14a0a`, then rerun the focused checks. -2. Push/open the PR from the main session. -3. Run the separately sized full-data build and calibration/k-ladder lane; - neither was executed here. +- PR #527 review cycle, then merge. After both #525 and #527: rebuild the + base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a + run that holds per `us_critical_targets.py`. +- Root record-level ETL carve stays open on populace#515. From a20e847852c3d09ae499b572b6840ed8732eac62 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 21:23:12 -0400 Subject: [PATCH 21/94] Bind the ported provenance schema in the multispine stage test The acs_transfer port added imputed_recipient_rows to AcsImputedInput; this pre-existing whole-dict assertion needed the new key (default 0 for the synthetic fixture). Co-Authored-By: Claude Fable 5 --- packages/populace-build/tests/test_us_acs_multispine.py | 1 + 1 file changed, 1 insertion(+) diff --git a/packages/populace-build/tests/test_us_acs_multispine.py b/packages/populace-build/tests/test_us_acs_multispine.py index cea91181..57362865 100644 --- a/packages/populace-build/tests/test_us_acs_multispine.py +++ b/packages/populace-build/tests/test_us_acs_multispine.py @@ -182,6 +182,7 @@ def fake_pool(actual_base, actual_acs, *, acs_share): "seed": 91, "weight_kind": "design", "patterns": [], + "imputed_recipient_rows": 0, "unmodeled_recipient_rows": 0, "derivation": None, "reconciliation": None, From b6c3046aaa70d74b5df571fa14dceec2c9c776c2 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 21:41:48 -0400 Subject: [PATCH 22/94] Start PR 583 hold remediation journal --- PROGRESS.md | 56 +++++++++++++++++------------------------------------ 1 file changed, 18 insertions(+), 38 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index f3158fdd..02174c94 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,47 +2,27 @@ ## State -Populace #516 whole-row donor outlier screen is complete on -`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515 -interim carve merged as #525). The `puf_tax_detail` donor now drops tax units -whose grouped raw mortgage interest reaches $10M before the #515 carve -(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T -of phantom mortgage-interest mass), with the checkpoint schema bumped to v3 -so post-carve pre-screen checkpoints rebuild. +PR #583 round-1 HOLD remediation is in progress on +`multispine-pool-build-578`. The four blocking seams are the raw-stage producer +boundary, crash-safe pool publication, pandas string-read guard coverage, and +the deprecated-but-supported local ACS release producer. ## Done -- Confirmed a clean starting worktree at `aef1c56`. -- Read the repository guidance and established the #515 donor carve as the - screen's required downstream boundary. -- Started source-level audits of every donor-frame consumer, checkpoint - validation, row-count pins, and existing donor-fact summaries. -- Attempted the requested GitNexus impact workflow; the managed filesystem - denied its global registry write. Its local index also exposed a broad - `build/` ignore mismatch, so the completed impact audit uses direct source - call sites and tests. -- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the - structural rationale and pinned-artifact receipts. -- Added a whole-row screen on grouped raw person `home_mortgage_interest` - after tax-unit assembly, before the #515 carve, with retained-index reset. -- Confirmed no downstream consumer pairs donor rows to the original HDF arrays - or carries a stale donor-length vector; values and weights always originate - from the same screened frame. -- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale - checkpoint regression track the live constant while retaining literal-v1 - corruptions. -- Added regression coverage for the exact grouped boundary, whole-row removal, - retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant. -- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets - 139; populace-data 138 with 1 skip. The directly affected tail-bound suite - adds 12 passes. Ruff format/check and `git diff --check` are clean. -- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line - audit, expected 208,611-row real-artifact effect, verification results, count - sweep, and deliberately untouched surfaces. +- Confirmed the worktree starts clean at review head `a20e847`. +- Read `CLAUDE.md` and the full adversarial review at + `_buildo-runtime/reviews/sol_583.log`. +- Selected the GitNexus debugging and impact-analysis workflows for lineage + tracing and blast-radius checks. +- Established the required final constraint: this live checkpoint will be + restored byte-for-byte to `origin/main` before handoff so no root journal + ships in the PR diff. ## Next -- PR #527 review cycle, then merge. After both #525 and #527: rebuild the - base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a - run that holds per `us_critical_targets.py`. -- Root record-level ETL carve stays open on populace#515. +- Trace producer/pool/release call graphs and artifact contracts. +- Add each reviewer repro as a failing regression before its implementation + fix. +- Fix and commit the four blockers in coherent steps. +- Run the full `populace-build` suite and Ruff, write the external handoff, and + restore `PROGRESS.md` to `origin/main`. From 884c2555546105fcef6fd5540bccca7421d9e9b2 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 21:46:54 -0400 Subject: [PATCH 23/94] Catch pandas string reads in spine-blindness guard --- .../tests/test_us_spine_blindness.py | 97 +++++++++++++++++++ 1 file changed, 97 insertions(+) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 4337be6a..4c93a585 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -4,6 +4,7 @@ import ast import fnmatch +import re from pathlib import Path _REPOSITORY_ROOT = Path(__file__).resolve().parents[3] @@ -321,6 +322,36 @@ def _subscript_source_expression( ) +def _pandas_expression_source(node: ast.AST) -> str | None: + """Resolve guarded column-shaped tokens in a static pandas expression.""" + + shape = _string_shape(node) + if shape is None: + return None + for token in re.findall(r"[\w*?\[\]-]+", shape): + if _is_source_column_shape(token): + return f"source column {token!r}" + return None + + +def _call_argument( + node: ast.Call, + *, + position: int, + keyword: str, +) -> ast.AST | None: + if len(node.args) > position: + return node.args[position] + return next( + ( + candidate.value + for candidate in node.keywords + if candidate.arg == keyword + ), + None, + ) + + def _assigned_names(target: ast.AST) -> tuple[str, ...]: if isinstance(target, ast.Name): return (target.id,) @@ -422,6 +453,32 @@ def visit_Call(self, node: ast.Call) -> None: attribute = self._expression(node.args[1]) if attribute is not None: self._record(node, f"getattr using {attribute}") + elif isinstance(node.func, ast.Attribute) and name == "get": + key = _call_argument(node, position=0, keyword="key") + if key is not None: + column = _subscript_source_expression( + key, + bindings=self.bindings, + factory_aliases=self.factory_aliases, + ) + if column is not None: + self._record(node, f".get() using {column}") + elif isinstance(node.func, ast.Attribute) and name in {"query", "eval"}: + expression = _call_argument(node, position=0, keyword="expr") + if expression is not None: + column = _pandas_expression_source(expression) + if column is not None: + self._record(node, f".{name}() using {column}") + elif isinstance(node.func, ast.Attribute) and name == "filter": + items = _call_argument(node, position=0, keyword="items") + if items is not None: + column = _subscript_source_expression( + items, + bindings=self.bindings, + factory_aliases=self.factory_aliases, + ) + if column is not None: + self._record(node, f".filter(items=...) using {column}") self.generic_visit(node) def visit_Attribute(self, node: ast.Attribute) -> None: @@ -777,6 +834,46 @@ def op(df): assert _source_spine_accesses(raw_spine_source_id) +def test_source_spine_ast_guard_detects_pandas_string_reads() -> None: + """Pandas string-based column APIs cannot bypass the structural guard.""" + + reviewer_get = """ +def op(df): + return df.get("person_support_channel") +""" + reviewer_query = """ +def op(df): + return df.query('person_support_channel == "acs"') +""" + pandas_eval = """ +def op(df): + return df.eval("person_spine_source_id == 1") +""" + pandas_filter = """ +def op(df): + return df.filter(items=["person_spine"]) +""" + pandas_loc = """ +def op(df): + return df.loc[..., "person_support_channel"] +""" + benign_reads = """ +def op(df): + df.get("age") + df.query("age >= 18") + df.eval("age + 1") + df.filter(items=["age"]) + return df.loc[..., "age"] +""" + + assert _source_spine_accesses(reviewer_get) + assert _source_spine_accesses(reviewer_query) + assert _source_spine_accesses(pandas_eval) + assert _source_spine_accesses(pandas_filter) + assert _source_spine_accesses(pandas_loc) + assert _source_spine_accesses(benign_reads) == () + + def test_source_spine_ast_guard_covers_every_entity_grain() -> None: """Every US and benefit-unit grain's source identity is prohibited.""" From 957ea8edefbdf05bcb4a27e29bae1eed7d7e036d Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 21:47:08 -0400 Subject: [PATCH 24/94] Checkpoint pandas guard remediation --- PROGRESS.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 02174c94..7300b709 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -14,15 +14,22 @@ the deprecated-but-supported local ACS release producer. `_buildo-runtime/reviews/sol_583.log`. - Selected the GitNexus debugging and impact-analysis workflows for lineage tracing and blast-radius checks. +- Attempted to build the GitNexus index; the managed lane denied its global + registry write, so direct source/call-site/test tracing is the active + fallback. Removed the generated untracked local cache. +- Extended the spine-blindness visitor over pandas string APIs and committed + the exact reviewer `.get()` and `.query()` mutations plus `.eval()`, + `.filter(items=...)`, and `.loc[...]` regressions in `884c255`. The focused + guard suite passes (11 tests) and Ruff is clean. - Established the required final constraint: this live checkpoint will be restored byte-for-byte to `origin/main` before handoff so no root journal ships in the PR diff. ## Next -- Trace producer/pool/release call graphs and artifact contracts. +- Finish tracing the producer/pool and legacy-release artifact contracts. - Add each reviewer repro as a failing regression before its implementation fix. -- Fix and commit the four blockers in coherent steps. +- Fix and commit the remaining three blockers in coherent steps. - Run the full `populace-build` suite and Ruff, write the external handoff, and restore `PROGRESS.md` to `origin/main`. From b75f352db3d2eed92cf5ee806c5989cfc97a1d2c Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 21:50:30 -0400 Subject: [PATCH 25/94] Make multispine publication interruption-safe --- .../src/populace/build/us_runtime/h5_io.py | 33 +- .../tests/test_us_multispine_pool_tool.py | 288 ++++++++++++++++-- tools/build_us_multispine_pool.py | 91 +++++- 3 files changed, 361 insertions(+), 51 deletions(-) diff --git a/packages/populace-build/src/populace/build/us_runtime/h5_io.py b/packages/populace-build/src/populace/build/us_runtime/h5_io.py index c30edd01..320afe01 100644 --- a/packages/populace-build/src/populace/build/us_runtime/h5_io.py +++ b/packages/populace-build/src/populace/build/us_runtime/h5_io.py @@ -74,6 +74,7 @@ def write_nullable_us_h5( *, period: int, artifact_kind: str, + publication_run_id: str | None = None, ) -> None: """Atomically write and verify a nullable US single-year H5. @@ -87,6 +88,10 @@ def write_nullable_us_h5( raise TypeError(f"frame must be a Frame, got {type(frame).__name__}.") if not isinstance(artifact_kind, str) or not artifact_kind.strip(): raise ValueError("artifact_kind must be a non-empty string.") + if publication_run_id is not None and ( + not isinstance(publication_run_id, str) or not publication_run_id.strip() + ): + raise ValueError("publication_run_id must be a non-empty string when set.") output = Path(path) output.parent.mkdir(parents=True, exist_ok=True) @@ -97,12 +102,14 @@ def write_nullable_us_h5( temporary, period=int(period), artifact_kind=artifact_kind, + publication_run_id=publication_run_id, ) _verify_nullable_us_h5( frame, temporary, period=int(period), artifact_kind=artifact_kind, + publication_run_id=publication_run_id, ) os.replace(temporary, output) except BaseException: @@ -116,6 +123,7 @@ def _write_nullable_us_h5_file( *, period: int, artifact_kind: str, + publication_run_id: str | None, ) -> None: with pd.HDFStore(path, mode="w") as store: for entity in frame.entities: @@ -138,7 +146,11 @@ def _write_nullable_us_h5_file( pd.Series( [ json.dumps( - _artifact_metadata(frame, artifact_kind=artifact_kind), + _artifact_metadata( + frame, + artifact_kind=artifact_kind, + publication_run_id=publication_run_id, + ), sort_keys=True, ) ] @@ -153,6 +165,7 @@ def _verify_nullable_us_h5( *, period: int, artifact_kind: str, + publication_run_id: str | None, ) -> None: with pd.HDFStore(path, mode="r") as store: for entity in frame.entities: @@ -197,7 +210,11 @@ def _verify_nullable_us_h5( raise RuntimeError( "Nullable US H5 artifact metadata is not valid JSON." ) from exc - expected_metadata = _artifact_metadata(frame, artifact_kind=artifact_kind) + expected_metadata = _artifact_metadata( + frame, + artifact_kind=artifact_kind, + publication_run_id=publication_run_id, + ) if stored_metadata != expected_metadata: raise RuntimeError( "Nullable US H5 round trip changed artifact metadata: " @@ -214,9 +231,17 @@ def _export_table(frame: Frame, entity: str) -> pd.DataFrame: return household -def _artifact_metadata(frame: Frame, *, artifact_kind: str) -> dict[str, str]: - return { +def _artifact_metadata( + frame: Frame, + *, + artifact_kind: str, + publication_run_id: str | None, +) -> dict[str, str]: + metadata = { "artifact_kind": artifact_kind, "entity_hdf_format": "fixed_nullable", "household_weight_kind": frame.weights_for("household").kind.value, } + if publication_run_id is not None: + metadata["publication_run_id"] = publication_run_id + return metadata diff --git a/packages/populace-build/tests/test_us_multispine_pool_tool.py b/packages/populace-build/tests/test_us_multispine_pool_tool.py index 06fc973a..e8b672ab 100644 --- a/packages/populace-build/tests/test_us_multispine_pool_tool.py +++ b/packages/populace-build/tests/test_us_multispine_pool_tool.py @@ -209,6 +209,95 @@ def simulate(person: pd.DataFrame) -> None: return result +def _output_context( + pool_tool: ModuleType, + tmp_path: Path, +): + result = _red_pool_result(pool_tool, tmp_path) + outputs = pool_tool._output_paths(tmp_path / "pool.h5") + source_manifest = pool_tool.load_acs_source_manifest() + verified_inputs = {} + for index, role in enumerate( + ( + "asec_pre_clone", + "acs_household", + "acs_person", + "processed_puf", + "puf_source_year", + ), + start=1, + ): + path = tmp_path / f"{role}.fixture" + path.write_bytes(f"input-{index}".encode()) + digest = hashlib.sha256(path.read_bytes()).hexdigest() + verified_inputs[role] = pool_tool._VerifiedInput( + role=role, + path=path, + expected_sha256=digest, + actual_sha256=digest, + size_bytes=path.stat().st_size, + ) + loaded = pool_tool._LoadedInputs( + asec=_source_frame(), + acs=_source_frame(measured_offset=99.0), + puf_donor=pd.DataFrame({"RECID": [1]}), + asec_checkpoint={"artifact": "fixture-pre-clone"}, + acs_build={"artifact": "fixture-unit-frame"}, + acs_native_inputs={"person": {"age": {"source": "fixture"}}}, + puf_donor_build={"artifact": "fixture-donor"}, + ) + return result, outputs, verified_inputs, source_manifest, loaded + + +def _seed_stale_green_outputs(outputs) -> None: + outputs.pool_h5.write_bytes(b"stale green h5") + outputs.agreement_diagnostics.write_text( + json.dumps( + { + "simulation_ready": True, + "publication_run_id": "stale-run", + } + ), + encoding="utf-8", + ) + outputs.manifest.write_text( + json.dumps( + { + "status": "simulation_ready", + "simulation_ready": True, + "publication_run_id": "stale-run", + } + ), + encoding="utf-8", + ) + + +def _assert_publication_tombstone( + pool_tool: ModuleType, + outputs, + *, + publication_run_id: str, +) -> None: + manifest = json.loads(outputs.manifest.read_text(encoding="utf-8")) + assert manifest == { + "agreement_diagnostics": { + "path": str(outputs.agreement_diagnostics.resolve()), + "publication_run_id": publication_run_id, + }, + "artifact_kind": "populace_us_multispine_pool_manifest", + "message": "publication in progress", + "pool_h5": { + "artifact_kind": "populace_us_multispine_input_pool", + "path": str(outputs.pool_h5.resolve()), + "publication_run_id": publication_run_id, + }, + "publication_run_id": publication_run_id, + "schema_version": pool_tool.POOL_MANIFEST_SCHEMA_VERSION, + "simulation_ready": False, + "status": "publication_in_progress", + } + + def test_parser_exposes_only_five_pinned_inputs_and_out( pool_tool: ModuleType, ) -> None: @@ -418,39 +507,9 @@ def test_red_outputs_preserve_receipts_and_exclude_simulation_output( tmp_path: Path, ) -> None: pytest.importorskip("tables") - result = _red_pool_result(pool_tool, tmp_path) - outputs = pool_tool._output_paths(tmp_path / "pool.h5") - source_manifest = pool_tool.load_acs_source_manifest() - - verified_inputs = {} - for index, role in enumerate( - ( - "asec_pre_clone", - "acs_household", - "acs_person", - "processed_puf", - "puf_source_year", - ), - start=1, - ): - path = tmp_path / f"{role}.fixture" - path.write_bytes(f"input-{index}".encode()) - digest = hashlib.sha256(path.read_bytes()).hexdigest() - verified_inputs[role] = pool_tool._VerifiedInput( - role=role, - path=path, - expected_sha256=digest, - actual_sha256=digest, - size_bytes=path.stat().st_size, - ) - loaded = pool_tool._LoadedInputs( - asec=_source_frame(), - acs=_source_frame(measured_offset=99.0), - puf_donor=pd.DataFrame({"RECID": [1]}), - asec_checkpoint={"artifact": "fixture-pre-clone"}, - acs_build={"artifact": "fixture-unit-frame"}, - acs_native_inputs={"person": {"age": {"source": "fixture"}}}, - puf_donor_build={"artifact": "fixture-donor"}, + result, outputs, verified_inputs, source_manifest, loaded = _output_context( + pool_tool, + tmp_path, ) pool_tool._write_outputs( @@ -475,14 +534,177 @@ def test_red_outputs_preserve_receipts_and_exclude_simulation_output( assert manifest["agreement_gate"] == expected_gate assert diagnostics["agreement_gate"] == expected_gate assert diagnostics["simulation_ready"] is False + assert diagnostics["publication_run_id"] == manifest["publication_run_id"] assert manifest["provenance_pins"] == { role: pin.to_manifest() for role, pin in verified_inputs.items() } assert manifest["pool_h5"]["formula_outputs_persisted"] is False assert manifest["pool_h5"]["input_only"] is True + assert ( + manifest["pool_h5"]["publication_run_id"] + == manifest["publication_run_id"] + ) + assert manifest["pool_h5"]["sha256"] == hashlib.sha256( + outputs.pool_h5.read_bytes() + ).hexdigest() with pd.HDFStore(outputs.pool_h5, mode="r") as store: assert "ssi" not in store["person"].columns + metadata = json.loads(str(store["_populace_staging_metadata"].iloc[0])) + assert metadata["publication_run_id"] == manifest["publication_run_id"] + + +def test_h5_publication_failure_invalidates_stale_green_manifest( + pool_tool: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + result, outputs, verified_inputs, source_manifest, loaded = _output_context( + pool_tool, + tmp_path, + ) + _seed_stale_green_outputs(outputs) + publication_run_id = "h5-failure-run" + monkeypatch.setattr( + pool_tool, + "_new_publication_run_id", + lambda: publication_run_id, + ) + + def fail_h5(*_args, **_kwargs) -> None: + _assert_publication_tombstone( + pool_tool, + outputs, + publication_run_id=publication_run_id, + ) + raise RuntimeError("injected H5 publication failure") + + monkeypatch.setattr(pool_tool, "write_nullable_us_h5", fail_h5) + + with pytest.raises(RuntimeError, match="injected H5 publication failure"): + pool_tool._write_outputs( + result, + outputs=outputs, + verified_inputs=verified_inputs, + acs_source_manifest=source_manifest, + loaded=loaded, + ) + + _assert_publication_tombstone( + pool_tool, + outputs, + publication_run_id=publication_run_id, + ) + assert outputs.pool_h5.read_bytes() == b"stale green h5" + + +def test_diagnostics_publication_failure_keeps_pool_not_ready( + pool_tool: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + result, outputs, verified_inputs, source_manifest, loaded = _output_context( + pool_tool, + tmp_path, + ) + _seed_stale_green_outputs(outputs) + publication_run_id = "diagnostics-failure-run" + monkeypatch.setattr( + pool_tool, + "_new_publication_run_id", + lambda: publication_run_id, + ) + temporary_diagnostics = pool_tool._publication_temporary_path( + outputs.agreement_diagnostics, + publication_run_id=publication_run_id, + ) + atomic_write_json = pool_tool._atomic_write_json + + def fail_diagnostics(path, payload) -> None: + if Path(path) == temporary_diagnostics: + raise RuntimeError("injected diagnostics publication failure") + atomic_write_json(path, payload) + + monkeypatch.setattr(pool_tool, "_atomic_write_json", fail_diagnostics) + + with pytest.raises( + RuntimeError, + match="injected diagnostics publication failure", + ): + pool_tool._write_outputs( + result, + outputs=outputs, + verified_inputs=verified_inputs, + acs_source_manifest=source_manifest, + loaded=loaded, + ) + + _assert_publication_tombstone( + pool_tool, + outputs, + publication_run_id=publication_run_id, + ) + assert outputs.pool_h5.read_bytes() == b"stale green h5" + assert not pool_tool._publication_temporary_path( + outputs.pool_h5, + publication_run_id=publication_run_id, + ).exists() + + +def test_final_manifest_failure_leaves_tombstone_as_readiness_authority( + pool_tool: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + result, outputs, verified_inputs, source_manifest, loaded = _output_context( + pool_tool, + tmp_path, + ) + _seed_stale_green_outputs(outputs) + publication_run_id = "manifest-failure-run" + monkeypatch.setattr( + pool_tool, + "_new_publication_run_id", + lambda: publication_run_id, + ) + atomic_write_json = pool_tool._atomic_write_json + + def fail_final_manifest(path, payload) -> None: + if ( + Path(path) == outputs.manifest + and payload["status"] != "publication_in_progress" + ): + raise RuntimeError("injected final manifest publication failure") + atomic_write_json(path, payload) + + monkeypatch.setattr(pool_tool, "_atomic_write_json", fail_final_manifest) + + with pytest.raises( + RuntimeError, + match="injected final manifest publication failure", + ): + pool_tool._write_outputs( + result, + outputs=outputs, + verified_inputs=verified_inputs, + acs_source_manifest=source_manifest, + loaded=loaded, + ) + + _assert_publication_tombstone( + pool_tool, + outputs, + publication_run_id=publication_run_id, + ) + diagnostics = json.loads( + outputs.agreement_diagnostics.read_text(encoding="utf-8") + ) + assert diagnostics["publication_run_id"] == publication_run_id + with pd.HDFStore(outputs.pool_h5, mode="r") as store: + metadata = json.loads(str(store["_populace_staging_metadata"].iloc[0])) + assert metadata["publication_run_id"] == publication_run_id def test_clone_safe_id_error_surfaces_unchanged_through_tool( diff --git a/tools/build_us_multispine_pool.py b/tools/build_us_multispine_pool.py index 7b061c76..b4078dd0 100644 --- a/tools/build_us_multispine_pool.py +++ b/tools/build_us_multispine_pool.py @@ -88,7 +88,7 @@ "main", ] -POOL_MANIFEST_SCHEMA_VERSION = 1 +POOL_MANIFEST_SCHEMA_VERSION = 2 """Schema version for the companion pool build manifest.""" POOL_H5_ARTIFACT_KIND = "populace_us_multispine_input_pool" @@ -568,6 +568,7 @@ def _manifest_payload( verified_inputs: Mapping[str, _VerifiedInput], acs_source_manifest: AcsSourceManifest, loaded: _LoadedInputs, + publication_run_id: str, ) -> dict[str, object]: status = "simulation_ready" if result.simulation_ready else "agreement_failed" return { @@ -575,6 +576,7 @@ def _manifest_payload( "schema_version": POOL_MANIFEST_SCHEMA_VERSION, "status": status, "simulation_ready": result.simulation_ready, + "publication_run_id": publication_run_id, "calibration_applied": False, "operator_order": list(POOL_OPERATOR_ORDER), "period": POOL_TIME_PERIOD, @@ -611,6 +613,7 @@ def _manifest_payload( "sha256": _file_sha256(outputs.pool_h5), "size_bytes": outputs.pool_h5.stat().st_size, "artifact_kind": POOL_H5_ARTIFACT_KIND, + "publication_run_id": publication_run_id, "nullable": True, "input_only": True, "formula_outputs_persisted": False, @@ -630,6 +633,41 @@ def _manifest_payload( } +def _publication_tombstone( + outputs: PoolBuildOutputs, + *, + publication_run_id: str, +) -> dict[str, object]: + return { + "artifact_kind": "populace_us_multispine_pool_manifest", + "schema_version": POOL_MANIFEST_SCHEMA_VERSION, + "status": "publication_in_progress", + "simulation_ready": False, + "publication_run_id": publication_run_id, + "message": "publication in progress", + "pool_h5": { + "path": str(outputs.pool_h5.resolve()), + "artifact_kind": POOL_H5_ARTIFACT_KIND, + "publication_run_id": publication_run_id, + }, + "agreement_diagnostics": { + "path": str(outputs.agreement_diagnostics.resolve()), + "publication_run_id": publication_run_id, + }, + } + + +def _new_publication_run_id() -> str: + return uuid.uuid4().hex + + +def _publication_temporary_path(path: Path, *, publication_run_id: str) -> Path: + output = Path(path) + return output.with_name( + f".{output.name}.{publication_run_id}.publication.tmp" + ) + + def _write_outputs( result: MultispinePoolResult, *, @@ -638,27 +676,52 @@ def _write_outputs( acs_source_manifest: AcsSourceManifest, loaded: _LoadedInputs, ) -> None: - write_nullable_us_h5( - result.frame, + publication_run_id = _new_publication_run_id() + _atomic_write_json( + outputs.manifest, + _publication_tombstone( + outputs, + publication_run_id=publication_run_id, + ), + ) + temporary_h5 = _publication_temporary_path( outputs.pool_h5, - period=POOL_TIME_PERIOD, - artifact_kind=POOL_H5_ARTIFACT_KIND, + publication_run_id=publication_run_id, + ) + temporary_diagnostics = _publication_temporary_path( + outputs.agreement_diagnostics, + publication_run_id=publication_run_id, ) diagnostics = { "artifact_kind": "populace_us_multispine_agreement_diagnostics", "schema_version": POOL_MANIFEST_SCHEMA_VERSION, "simulation_ready": result.simulation_ready, + "publication_run_id": publication_run_id, "agreement_gate": _agreement_payload(result), } - _atomic_write_json(outputs.agreement_diagnostics, diagnostics) - manifest = _manifest_payload( - result=result, - outputs=outputs, - verified_inputs=verified_inputs, - acs_source_manifest=acs_source_manifest, - loaded=loaded, - ) - _atomic_write_json(outputs.manifest, manifest) + try: + write_nullable_us_h5( + result.frame, + temporary_h5, + period=POOL_TIME_PERIOD, + artifact_kind=POOL_H5_ARTIFACT_KIND, + publication_run_id=publication_run_id, + ) + _atomic_write_json(temporary_diagnostics, diagnostics) + os.replace(temporary_h5, outputs.pool_h5) + os.replace(temporary_diagnostics, outputs.agreement_diagnostics) + manifest = _manifest_payload( + result=result, + outputs=outputs, + verified_inputs=verified_inputs, + acs_source_manifest=acs_source_manifest, + loaded=loaded, + publication_run_id=publication_run_id, + ) + _atomic_write_json(outputs.manifest, manifest) + finally: + temporary_h5.unlink(missing_ok=True) + temporary_diagnostics.unlink(missing_ok=True) def _file_sha256(path: Path) -> str: From 58de86f7a0e2a491e57437e4568f7433231a7378 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 21:50:48 -0400 Subject: [PATCH 26/94] Checkpoint interruption-safe publication --- PROGRESS.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/PROGRESS.md b/PROGRESS.md index 7300b709..8126c69f 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -21,6 +21,10 @@ the deprecated-but-supported local ACS release producer. the exact reviewer `.get()` and `.query()` mutations plus `.eval()`, `.filter(items=...)`, and `.loc[...]` regressions in `884c255`. The focused guard suite passes (11 tests) and Ruff is clean. +- Reworked pool publication to invalidate first, stage H5 and diagnostics + under one run ID, publish them atomically, and write the readiness manifest + last in `b75f352`. The three reviewer interruption points now retain a + non-ready tombstone; focused H5/pool coverage passes (17 tests). - Established the required final constraint: this live checkpoint will be restored byte-for-byte to `origin/main` before handoff so no root journal ships in the PR diff. @@ -30,6 +34,6 @@ the deprecated-but-supported local ACS release producer. - Finish tracing the producer/pool and legacy-release artifact contracts. - Add each reviewer repro as a failing regression before its implementation fix. -- Fix and commit the remaining three blockers in coherent steps. +- Fix and commit the raw-boundary and legacy-shim blockers. - Run the full `populace-build` suite and Ruff, write the external handoff, and restore `PROGRESS.md` to `origin/main`. From f8bcb3b634a3f33699b6f0e74ccee3ad7342ca92 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 21:52:36 -0400 Subject: [PATCH 27/94] Restore deprecated ACS local-release builder --- .../tests/test_us_acs_local_release_tool.py | 46 + .../test_us_acs_multispine_base_builder.py | 41 +- .../test_us_acs_multispine_legacy_builder.py | 1043 +++++++++++++++ tools/_legacy/build_us_acs_multispine_base.py | 1173 +++++++++++++++++ tools/build_us_acs_local_release.py | 17 +- tools/build_us_acs_multispine_base.py | 63 +- 6 files changed, 2343 insertions(+), 40 deletions(-) create mode 100644 packages/populace-build/tests/test_us_acs_multispine_legacy_builder.py create mode 100644 tools/_legacy/build_us_acs_multispine_base.py diff --git a/packages/populace-build/tests/test_us_acs_local_release_tool.py b/packages/populace-build/tests/test_us_acs_local_release_tool.py index e7adc8eb..c8b38c91 100644 --- a/packages/populace-build/tests/test_us_acs_local_release_tool.py +++ b/packages/populace-build/tests/test_us_acs_local_release_tool.py @@ -4,6 +4,7 @@ import importlib.util import json +import shlex from pathlib import Path import numpy as np @@ -24,6 +25,19 @@ def _load_tool_module(): return module +def _load_staging_builder_module(): + root = Path(__file__).resolve().parents[3] + path = root / "tools" / "build_us_acs_multispine_base.py" + spec = importlib.util.spec_from_file_location( + "build_us_acs_multispine_base", + path, + ) + module = importlib.util.module_from_spec(spec) + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + def test_spine_composition_reports_per_spine_weight_and_size() -> None: module = _load_tool_module() households = pd.DataFrame( @@ -157,6 +171,38 @@ def test_release_id_prefix_and_manifest_constants() -> None: assert module.HF_REPO_ID == "policyengine/populace-us" +def test_documented_staging_recipe_matches_legacy_and_release_parsers( + tmp_path: Path, +) -> None: + release = _load_tool_module() + staging_builder = _load_staging_builder_module() + recipe = shlex.split(release.LEGACY_STAGING_REFRESH_RECIPE) + + assert recipe[:3] == [ + "uv", + "run", + "tools/build_us_acs_multispine_base.py", + ] + staging_args = staging_builder._legacy._parse_args(recipe[3:]) + staging_summary = staging_args.summary or staging_args.out_h5.with_suffix( + ".summary.json" + ) + release_args = release._parse_args( + [ + "--stage", + "materialize", + "--staging-h5", + str(staging_args.out_h5), + "--checkpoint-dir", + str(tmp_path / "ckpt"), + "--feed", + str(tmp_path / "facts.jsonl"), + ] + ) + + assert release._staging_summary_path(release_args) == staging_summary + + def test_do_finalize_requires_calibration_diagnostics(tmp_path: Path) -> None: module = _load_tool_module() staging = tmp_path / "staging.h5" diff --git a/packages/populace-build/tests/test_us_acs_multispine_base_builder.py b/packages/populace-build/tests/test_us_acs_multispine_base_builder.py index 87f905cd..e0931d71 100644 --- a/packages/populace-build/tests/test_us_acs_multispine_base_builder.py +++ b/packages/populace-build/tests/test_us_acs_multispine_base_builder.py @@ -203,27 +203,40 @@ def __init__(self, *, file_path: str) -> None: assert "household_weight" not in loaded.table("household") -def test_retired_cli_exits_nonzero_with_pool_builder_migration_command( - tmp_path: Path, -) -> None: - output = tmp_path / "must-not-exist.h5" - +def test_deprecated_cli_keeps_legacy_staging_recipe_available() -> None: result = subprocess.run( [ sys.executable, str(_shim_path()), - "--base-h5", - "legacy.h5", - "--out-h5", - str(output), + "--help", ], check=False, capture_output=True, text=True, ) - assert result.returncode == 2 - assert result.stdout == "" - assert "legacy command cannot be translated" in result.stderr - assert "uv run tools/build_us_multispine_pool.py --help" in result.stderr - assert not output.exists() + assert result.returncode == 0 + assert "--base-h5 BASE_H5" in result.stdout + assert "--out-h5 OUT_H5" in result.stdout + assert "--donor-release-manifest DONOR_RELEASE_MANIFEST" in result.stdout + assert "populace#578 increment 4" in result.stderr + assert "tools/build_us_multispine_pool.py" in result.stderr + + +def test_deprecated_shim_warns_and_delegates_arguments( + monkeypatch: pytest.MonkeyPatch, +) -> None: + shim = _load_shim_module() + argv = ["--base-h5", "dense.h5", "--out-h5", "staging.h5"] + captured: list[list[str] | None] = [] + + def fake_main(actual_argv: list[str] | None = None) -> int: + captured.append(actual_argv) + return 0 + + monkeypatch.setattr(shim._legacy, "main", fake_main) + + with pytest.warns(DeprecationWarning, match=r"populace#578 increment 4"): + assert shim.main(argv) == 0 + + assert captured == [argv] diff --git a/packages/populace-build/tests/test_us_acs_multispine_legacy_builder.py b/packages/populace-build/tests/test_us_acs_multispine_legacy_builder.py new file mode 100644 index 00000000..f8cba315 --- /dev/null +++ b/packages/populace-build/tests/test_us_acs_multispine_legacy_builder.py @@ -0,0 +1,1043 @@ +from __future__ import annotations + +import hashlib +import importlib.util +import json +from pathlib import Path +from types import SimpleNamespace + +import numpy as np +import pandas as pd +import pytest + +from populace.build import FitWeightRecord +from populace.build.us_runtime.acs_sources import ( + AcsSourceArtifact, + AcsSourceManifest, +) +from populace.build.us_runtime.base_pool import spine_column +from populace.build.us_runtime.puf_support import support_channel_column +from populace.frame import US_SCHEMA, Frame, WeightKind, Weights + + +def _load_builder_module(): + root = Path(__file__).resolve().parents[3] + path = root / "tools" / "_legacy" / "build_us_acs_multispine_base.py" + spec = importlib.util.spec_from_file_location( + "legacy_build_us_acs_multispine_base", + path, + ) + module = importlib.util.module_from_spec(spec) + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +def _frame( + *, + benefit_participation: bool = True, + spines: tuple[str, str] | None = None, +) -> Frame: + person = pd.DataFrame( + { + "person_id": [1, 2], + "person_household_id": [1, 2], + "person_tax_unit_id": [1, 2], + "person_spm_unit_id": [1, 2], + "person_family_id": [1, 2], + "person_marital_unit_id": [1, 2], + "age": [35, 67], + "is_female": [False, True], + } + ) + if benefit_participation: + person["takes_up_snap_if_eligible"] = [True, False] + tables = { + "person": person, + "household": pd.DataFrame({"household_id": [1, 2], "state_fips": [6, 36]}), + "tax_unit": pd.DataFrame({"tax_unit_id": [1, 2]}), + "spm_unit": pd.DataFrame({"spm_unit_id": [1, 2]}), + "family": pd.DataFrame({"family_id": [1, 2]}), + "marital_unit": pd.DataFrame({"marital_unit_id": [1, 2]}), + } + if spines is not None: + for entity, table in tables.items(): + table[spine_column(entity)] = list(spines) + return Frame( + tables, + US_SCHEMA, + { + "household": Weights( + np.asarray([40.0, 60.0]), + WeightKind.CALIBRATED, + ) + }, + ) + + +def _manifest() -> AcsSourceManifest: + source_directory = ( + "https://www2.census.gov/programs-surveys/acs/data/pums/2024/1-Year/" + ) + return AcsSourceManifest( + version=1, + spine="acs_2024_1yr", + vintage=2024, + verified_on="2026-07-10", + source_directory=source_directory, + artifacts=( + AcsSourceArtifact( + role="household", + filename="csv_hus.zip", + url=f"{source_directory}csv_hus.zip", + sha256="1" * 64, + size_bytes=123, + ), + AcsSourceArtifact( + role="person", + filename="csv_pus.zip", + url=f"{source_directory}csv_pus.zip", + sha256="2" * 64, + size_bytes=456, + ), + ), + ) + + +def test_parser_exposes_production_defaults_and_transfer_controls() -> None: + builder = _load_builder_module() + + args = builder._parse_args( + [ + "--base-h5", + "dense.h5", + "--out-h5", + "combined.h5", + ] + ) + + assert args.inputs_dir == builder.DEFAULT_INPUTS_DIR + assert args.inputs_dir.is_absolute() + assert args.inputs_dir.parts[-2:] == ("inputs", "acs_2024_1yr") + assert args.puma_ladder == builder.DEFAULT_PUMA_LADDER + assert args.puma_ladder.is_absolute() + assert args.n_estimators == 32 + assert args.max_targets_per_fit == 8 + assert args.period == 2024 + assert args.acs_share == 0.5 + assert args.seed == 0 + assert args.geography_seed == 0 + assert args.donor_channel == builder.ACS_DONOR_CHANNEL_AUTO + + custom = builder._parse_args( + [ + "--base-h5", + "dense.h5", + "--out-h5", + "combined.h5", + "--chunksize", + "1234", + "--acs-share", + "0.3", + "--seed", + "17", + "--geography-seed", + "23", + "--puma-ladder", + "ladder.npz", + "--n-estimators", + "9", + "--max-targets-per-fit", + "3", + "--donor-channel", + "benefit_support", + ] + ) + assert custom.chunksize == 1234 + assert custom.acs_share == 0.3 + assert custom.seed == 17 + assert custom.geography_seed == 23 + assert custom.puma_ladder == Path("ladder.npz") + assert custom.n_estimators == 9 + assert custom.max_targets_per_fit == 3 + assert custom.donor_channel == "benefit_support" + + +def test_main_wires_verified_sources_transfer_audit_export_and_summary( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + builder = _load_builder_module() + manifest = _manifest() + base = _frame() + combined = _frame(spines=("asec_puf", "acs_2024_1yr")) + combined.table("household")["puma"] = ["0100100", "0200100"] + combined.table("household")["congressional_district_geoid"] = [101, 200] + combined.table("household")["county_fips"] = ["01001", "02020"] + combined.table("household")["TYPEHUGQ"] = [1.0, 3.0] + base_h5 = tmp_path / "dense.h5" + base_h5.write_bytes(b"dense-base") + manifest_path = tmp_path / "acs_sources.json" + manifest_path.write_text("{}", encoding="utf-8") + puma_ladder_path = tmp_path / "us_puma_ladder_2020.npz" + puma_ladder_path.write_bytes(b"puma-ladder") + puma_ladder = builder.UsPumaLadder( + puma=np.asarray([100_100], dtype=np.int64), + puma_population=np.asarray([100.0]), + cd_overlap_puma=np.asarray([100_100], dtype=np.int64), + cd_overlap_cd=np.asarray([101], dtype=np.int64), + cd_overlap_population=np.asarray([100.0]), + county_overlap_puma=np.asarray([100_100], dtype=np.int64), + county_overlap_county=np.asarray([1_001], dtype=np.int32), + county_overlap_population=np.asarray([100.0]), + tract_overlap_puma=np.asarray([100_100], dtype=np.int64), + tract_overlap_tract=np.asarray([1_001_000_100], dtype=np.int64), + tract_overlap_population=np.asarray([100.0]), + metadata={ + "schema_version": 1, + "kind": "us_puma_ladder", + "puma_vintage": "2020_puma", + "sampling_basis": "population", + "layers": { + "congressional_district": {"vintage": "119th_congress"}, + "county": {"vintage": "2020_census"}, + "tract": {"vintage": "2020_census"}, + }, + }, + ) + inputs_dir = tmp_path / "inputs" + household_zip = inputs_dir / "csv_hus.zip" + person_zip = inputs_dir / "csv_pus.zip" + source = builder.AcsPumsSource(household_zip, person_zip) + output_h5 = tmp_path / "combined.h5" + summary_path = tmp_path / "combined.summary.json" + captured: dict[str, object] = {} + transfer_plan = { + "person": { + "benefit_participation": ("takes_up_snap_if_eligible",), + } + } + + def fake_load_manifest(path): + captured["manifest_path"] = path + return manifest + + def fake_fetch(cache_dir, *, manifest): + captured["fetch"] = (cache_dir, manifest) + return source + + def fake_build(actual_base, actual_source, **kwargs): + captured["build"] = (actual_base, actual_source, kwargs) + return builder.AcsMultispineResult( + frame=combined, + fit_records=(FitWeightRecord("acs_transfer:person:benefits", "design"),), + provenance={ + "enabled": True, + "deferred_inputs": ["block_geoid", "tract_geoid"], + "geography_ladder": { + "applied": True, + "household_rows": 2, + "ladder_pumas": 1, + "layer_vintages": puma_ladder.layer_vintages, + "sampling_basis": "population", + "seed": 19, + "resolved_model_inputs": [ + "congressional_district_geoid", + "county_fips", + ], + "unresolved_sub_puma_inputs": [ + "block_geoid", + "tract_geoid", + ], + }, + "imputed_inputs": [ + { + "column": "takes_up_snap_if_eligible", + "family": "benefit_participation", + "unmodeled_recipient_rows": 0, + } + ], + }, + ) + + def fake_write(frame, path, *, period): + captured["write"] = (frame, path, period) + path.write_bytes(b"combined-output") + + def fake_load_puma_ladder(path): + captured["puma_ladder_path"] = path + return puma_ladder + + reviewed_null = { + "entity": "person", + "column": "pre_subsidy_rent", + "dtype": "float", + "missing_rows": 1, + "rows": 2, + "missing_rows_by_spine": {"acs_2024_1yr": 1}, + } + + monkeypatch.setattr(builder, "_load_base_frame", lambda path: base) + monkeypatch.setattr( + builder, + "_require_dense_donor_coverage", + lambda frame, **kwargs: None, + ) + monkeypatch.setattr( + builder, + "declared_acs_transfer_target_families", + lambda: transfer_plan, + ) + monkeypatch.setattr( + builder.acs_sources, + "load_acs_source_manifest", + fake_load_manifest, + ) + monkeypatch.setattr( + builder.acs_sources, + "fetch_acs_pums_sources", + fake_fetch, + ) + monkeypatch.setattr(builder, "build_optional_acs_multispine", fake_build) + monkeypatch.setattr( + builder, + "load_us_puma_ladder", + fake_load_puma_ladder, + ) + monkeypatch.setattr( + builder, + "_engine_input_null_audit", + lambda frame: [reviewed_null], + ) + monkeypatch.setattr( + builder, + "_preflight_staging_export", + lambda frame: 123_456, + ) + monkeypatch.setattr(builder, "_write_dataset", fake_write) + + exit_code = builder.main( + [ + "--base-h5", + str(base_h5), + "--out-h5", + str(output_h5), + "--summary", + str(summary_path), + "--source-manifest", + str(manifest_path), + "--inputs-dir", + str(inputs_dir), + "--puma-ladder", + str(puma_ladder_path), + "--max-households", + "7", + "--chunksize", + "2000", + "--acs-share", + "0.4", + "--seed", + "11", + "--geography-seed", + "19", + ] + ) + + assert exit_code == 0 + assert captured["manifest_path"] == manifest_path + assert captured["puma_ladder_path"] == puma_ladder_path + assert captured["fetch"] == (inputs_dir, manifest) + actual_base, actual_source, build_options = captured["build"] + assert actual_base is base + assert actual_source.max_households == 7 + assert build_options.pop("puma_ladder") is puma_ladder + assert build_options == { + "chunksize": 2000, + "acs_share": 0.4, + "target_families": transfer_plan, + "donor_channel": builder.ACS_DONOR_CHANNEL_AUTO, + "seed": 11, + "n_estimators": 32, + "max_targets_per_fit": 8, + "geography_seed": 19, + } + assert captured["write"] == (combined, output_h5, 2024) + + summary = json.loads(summary_path.read_text(encoding="utf-8")) + assert summary["artifact_kind"] == "nullable_precalibration_staging_h5" + assert summary["calibration_applied"] is False + assert summary["simulation_ready"] is False + assert summary["simulation_ready_except_calibration"] is True + assert summary["simulation_readiness_blockers"] == ["calibration_not_applied"] + assert [item["id"] for item in summary["reviewed_limitations"]] == [ + "acs_group_quarters_housing_universe", + "native_acs_source_universe_blanks", + "sub_puma_geographic_precision", + ] + assert summary["reviewed_limitations"][2]["unavailable_exact_geography"] == [ + "block_geoid", + "tract_geoid", + ] + assert summary["reviewed_engine_input_nulls"] == [reviewed_null] + assert "pending_engine_input_nulls" not in summary + assert summary["staging_export_peak_estimate_bytes"] == 123_456 + assert summary["geography_ladder"] == { + "path": str(puma_ladder_path.resolve()), + "sha256": hashlib.sha256(b"puma-ladder").hexdigest(), + "pumas": 1, + "layer_vintages": { + "congressional_district": "119th_congress", + "county": "2020_census", + "puma": "2020_puma", + "tract": "2020_census", + }, + "seed": 19, + "assignment": summary["orchestration"]["provenance"]["geography_ladder"], + } + assert summary["acs_sources"]["manifest"] == str(manifest_path.resolve()) + assert ( + summary["acs_sources"]["manifest_sha256"] == hashlib.sha256(b"{}").hexdigest() + ) + assert summary["acs_sources"]["artifacts"] == [ + { + "role": "household", + "filename": "csv_hus.zip", + "url": manifest.artifacts[0].url, + "sha256": "1" * 64, + "size_bytes": 123, + "local_path": str(household_zip.resolve()), + }, + { + "role": "person", + "filename": "csv_pus.zip", + "url": manifest.artifacts[1].url, + "sha256": "2" * 64, + "size_bytes": 456, + "local_path": str(person_zip.resolve()), + }, + ] + assert summary["weights_audit"]["details"]["resolved_weight_kinds"] == { + "acs_transfer:person:benefits": "design" + } + assert summary["rows"]["combined"] == {entity: 2 for entity in US_SCHEMA.entities} + assert summary["household_weight_totals"] == { + "base": 100.0, + "combined": 100.0, + } + assert summary["spine_totals"] == { + "acs_2024_1yr": { + "rows": {entity: 1 for entity in US_SCHEMA.entities}, + "household_weight_total": 60.0, + }, + "asec_puf": { + "rows": {entity: 1 for entity in US_SCHEMA.entities}, + "household_weight_total": 40.0, + }, + } + assert summary["output"]["sha256"] == hashlib.sha256(b"combined-output").hexdigest() + + +def test_weights_audit_failure_aborts_before_export() -> None: + builder = _load_builder_module() + result = builder.AcsMultispineResult( + frame=_frame(), + fit_records=(FitWeightRecord("acs_transfer:person:benefits", "none"),), + ) + + with pytest.raises(SystemExit) as exc: + builder._audit_fits(result) + + assert "Weights audit failed" in str(exc.value) + assert "acs_transfer:person:benefits" in str(exc.value) + assert "unweighted" in str(exc.value) + + +def test_default_transfer_fails_closed_before_fetch_without_benefit_participation( + monkeypatch: pytest.MonkeyPatch, +) -> None: + builder = _load_builder_module() + monkeypatch.setattr( + builder.acs_sources, + "load_acs_source_manifest", + lambda path: _manifest(), + ) + monkeypatch.setattr( + builder, + "_load_base_frame", + lambda path: _frame(benefit_participation=False), + ) + monkeypatch.setattr(builder, "_sha256", lambda path: "0" * 64) + + def must_not_fetch(*_args, **_kwargs): + raise AssertionError("source fetch must not run for an incomplete donor") + + monkeypatch.setattr( + builder.acs_sources, + "fetch_acs_pums_sources", + must_not_fetch, + ) + + with pytest.raises(SystemExit) as exc: + builder.main( + [ + "--base-h5", + "incomplete.h5", + "--out-h5", + "combined.h5", + ] + ) + + assert "no takes_up_*" in str(exc.value) + assert "must run after the benefit input-family stages" in str(exc.value) + + +def test_empty_transfer_audit_is_not_treated_as_success() -> None: + builder = _load_builder_module() + + with pytest.raises(SystemExit) as exc: + builder._audit_fits( + SimpleNamespace( + fit_records=(), + ) + ) + + assert "produced no fit records" in str(exc.value) + + +class _DefaultsEngine: + _defaults = { + "age": 0, + "alimony_income": 0.0, + "employment_income_before_lsr": 0.0, + "has_esi": False, + "is_female": False, + "state_fips": 0, + "takes_up_snap_if_eligible": True, + "weekly_hours_worked_before_lsr": 40.0, + } + + def default_values(self, names): + return {name: self._defaults[name] for name in names if name in self._defaults} + + +def test_dense_donor_ignores_default_runtime_column_not_consumed_by_transfer() -> None: + builder = _load_builder_module() + base = _frame() + base.person["weekly_hours_worked_before_lsr"] = [40.0, 40.0] + + builder._require_dense_donor_coverage( + base, + engine=_DefaultsEngine(), + donor_channel=None, + target_families={ + "person": { + "benefit_participation": ("takes_up_snap_if_eligible",), + } + }, + ) + + +def test_dense_donor_missing_transfer_consumed_column_fails_hard() -> None: + builder = _load_builder_module() + + with pytest.raises(SystemExit) as exc: + builder._require_dense_donor_coverage( + _frame(), + engine=_DefaultsEngine(), + donor_channel=None, + target_families={"person": {"health": ("has_esi",)}}, + ) + + message = str(exc.value) + assert "hard ACS transfer-consumption gate" in message + assert "person.has_esi" in message + assert "transfer-consumed column is absent" in message + + +def test_dense_donor_default_transfer_consumed_column_fails_hard() -> None: + builder = _load_builder_module() + base = _frame() + base.person["has_esi"] = [False, False] + + with pytest.raises(SystemExit) as exc: + builder._require_dense_donor_coverage( + base, + engine=_DefaultsEngine(), + donor_channel=None, + target_families={"person": {"health": ("has_esi",)}}, + ) + + message = str(exc.value) + assert "hard ACS transfer-consumption gate" in message + assert "person.has_esi" in message + assert "every observed donor value equals the engine default" in message + + +def test_dense_donor_default_consumed_optional_feature_fails_hard() -> None: + builder = _load_builder_module() + base = _frame() + base.person["employment_income_before_lsr"] = [0.0, 0.0] + + with pytest.raises(SystemExit) as exc: + builder._require_dense_donor_coverage( + base, + engine=_DefaultsEngine(), + donor_channel=None, + target_families={ + "person": { + "benefit_participation": ("takes_up_snap_if_eligible",), + } + }, + ) + + message = str(exc.value) + assert "person.employment_income_before_lsr" in message + assert "every observed donor value equals the engine default" in message + + +def test_signal_bearing_release_exclusion_is_irrelevant_to_donor_gate( + capsys: pytest.CaptureFixture[str], +) -> None: + builder = _load_builder_module() + base = _frame() + base.person["alimony_income"] = [0.0, 100.0] + + builder._require_dense_donor_coverage( + base, + engine=_DefaultsEngine(), + donor_channel=None, + target_families={ + "person": { + "benefit_participation": ("takes_up_snap_if_eligible",), + } + }, + ) + + captured = capsys.readouterr() + assert captured.out == "" + assert captured.err == "" + + +def test_dense_donor_coverage_is_checked_on_resolved_puf_channel( + monkeypatch: pytest.MonkeyPatch, +) -> None: + builder = _load_builder_module() + base = _frame() + channel_column = support_channel_column("person") + for entity in base.entities: + base.table(entity)[support_channel_column(entity)] = [ + "asec", + "puf_tax_detail", + ] + captured: dict[str, Frame] = {} + + def fake_requirements(frame, target_families): + captured["frame"] = frame + return {"person": ("age",)} + + monkeypatch.setattr( + builder, + "acs_transfer_donor_requirements", + fake_requirements, + ) + + builder._require_dense_donor_coverage( + base, + engine=_DefaultsEngine(), + target_families={"person": {"test": ("age",)}}, + ) + + selected = captured["frame"] + assert selected.n("person") == 1 + assert selected.person[channel_column].tolist() == ["puf_tax_detail"] + + +def test_partial_donor_channel_metadata_fails_before_coverage_gate( + monkeypatch: pytest.MonkeyPatch, +) -> None: + builder = _load_builder_module() + base = _frame() + base.table("household")[support_channel_column("household")] = [ + "asec", + "puf_tax_detail", + ] + called = False + + def fake_requirements(frame, target_families): + nonlocal called + called = True + return {"person": ("age",)} + + monkeypatch.setattr( + builder, + "acs_transfer_donor_requirements", + fake_requirements, + ) + + with pytest.raises(SystemExit, match="partial support metadata"): + builder._require_dense_donor_coverage( + base, + engine=_DefaultsEngine(), + target_families={"person": {"test": ("age",)}}, + ) + + assert not called + + +def test_default_transfer_cannot_report_success_without_benefit_imputation() -> None: + builder = _load_builder_module() + result = builder.AcsMultispineResult( + frame=_frame(), + provenance={ + "imputed_inputs": [ + { + "column": "qualified_dividend_income", + "family": "puf_tax_itemization", + } + ] + }, + ) + + with pytest.raises(SystemExit) as exc: + builder._require_benefit_participation_transfer(result) + + assert "no takes_up_* benefit-participation input" in str(exc.value) + + +def test_default_transfer_must_register_every_planned_input( + monkeypatch: pytest.MonkeyPatch, +) -> None: + builder = _load_builder_module() + result = builder.AcsMultispineResult( + frame=_frame(), + provenance={ + "imputed_inputs": [ + { + "column": "takes_up_snap_if_eligible", + "family": "benefit_participation", + } + ] + }, + ) + monkeypatch.setattr( + builder, + "default_acs_transfer_target_families", + lambda donor: { + "person": { + "benefit_participation": ("takes_up_snap_if_eligible",), + "model_required": ("has_esi",), + } + }, + ) + + with pytest.raises(SystemExit) as exc: + builder._require_default_transfer_coverage(result, _frame()) + + assert "omitted donor-observed model input" in str(exc.value) + assert "has_esi" in str(exc.value) + + +def test_staging_h5_round_trips_base_only_nullable_boolean(tmp_path: Path) -> None: + pytest.importorskip("tables") # pandas HDF backend + builder = _load_builder_module() + frame = _frame(spines=("asec_puf", "acs_2024_1yr")) + frame.table("person")["is_snap_abawd_discretionary_exempt"] = pd.Series( + [True, np.nan], dtype=object + ) + output = tmp_path / "nullable-bool-staging.h5" + + builder._write_dataset(frame, output, period=2024) + + stored = pd.read_hdf(output, key="person") + assert stored.loc[0, "is_snap_abawd_discretionary_exempt"] is True + assert pd.isna(stored.loc[1, "is_snap_abawd_discretionary_exempt"]) + + +def test_engine_input_null_audit_includes_float_bool_and_spine_counts() -> None: + builder = _load_builder_module() + frame = _frame(spines=("asec_puf", "acs_2024_1yr")) + frame.person["age"] = [35.0, np.nan] + frame.person["takes_up_snap_if_eligible"] = pd.Series([True, np.nan], dtype=object) + + class FakeEngine: + def variables(self): + return ["age", "takes_up_snap_if_eligible"] + + def variable_metadata(self, name): + return SimpleNamespace( + dtype={ + "age": "float", + "takes_up_snap_if_eligible": "bool", + }[name] + ) + + audit = builder._engine_input_null_audit(frame, FakeEngine()) + + assert audit == [ + { + "entity": "person", + "column": "age", + "dtype": "float", + "missing_rows": 1, + "rows": 2, + "missing_rows_by_spine": {"acs_2024_1yr": 1}, + }, + { + "entity": "person", + "column": "takes_up_snap_if_eligible", + "dtype": "bool", + "missing_rows": 1, + "rows": 2, + "missing_rows_by_spine": {"acs_2024_1yr": 1}, + }, + ] + + +def test_nullable_staging_writer_round_trips_group_quarters_blanks( + tmp_path: Path, +) -> None: + pytest.importorskip("tables") # pandas HDF backend + builder = _load_builder_module() + frame = _frame(spines=("asec_puf", "acs_2024_1yr")) + frame.person["real_estate_taxes"] = [1_000.0, np.nan] + frame.person["pre_subsidy_rent"] = [1_200.0, np.nan] + frame.table("household")["TYPEHUGQ"] = [1.0, 3.0] + frame.table("household")["tenure_type"] = ["RENTED", np.nan] + frame.table("spm_unit")["spm_unit_tenure_type"] = ["RENTER", np.nan] + output = tmp_path / "staging.h5" + + builder._write_dataset(frame, output, period=2024) + + with pd.HDFStore(output, mode="r") as store: + assert store["person"]["real_estate_taxes"].isna().sum() == 1 + assert store["person"]["pre_subsidy_rent"].isna().sum() == 1 + assert store["household"]["tenure_type"].isna().sum() == 1 + assert store["spm_unit"]["spm_unit_tenure_type"].isna().sum() == 1 + assert store["household"]["household_weight"].tolist() == [40.0, 60.0] + metadata = json.loads(store["_populace_staging_metadata"].iloc[0]) + assert metadata == { + "artifact_kind": "nullable_precalibration_staging_h5", + "entity_hdf_format": "fixed_nullable", + "household_weight_kind": "calibrated", + } + + +def test_staging_export_preflight_enforces_decimal_30gb_limit() -> None: + builder = _load_builder_module() + frame = _frame() + + estimate = builder._preflight_staging_export(frame) + assert estimate < builder.DEFAULT_STAGING_EXPORT_PEAK_LIMIT_BYTES + with pytest.raises(MemoryError, match="above the 0.00 GB limit"): + builder._preflight_staging_export(frame, max_peak_bytes=1) + + +def test_registered_transfer_with_unmodeled_required_rows_fails( + monkeypatch: pytest.MonkeyPatch, +) -> None: + builder = _load_builder_module() + frame = _frame(spines=("asec_puf", "acs_2024_1yr")) + frame.person["has_esi"] = pd.Series([True, np.nan], dtype=object) + result = builder.AcsMultispineResult( + frame=frame, + provenance={ + "imputed_inputs": [ + { + "column": "has_esi", + "family": "model_required_boolean", + "unmodeled_recipient_rows": 1, + } + ] + }, + ) + monkeypatch.setattr( + builder, + "default_acs_transfer_target_families", + lambda donor: {"person": {"model_required_boolean": ("has_esi",)}}, + ) + + with pytest.raises(SystemExit, match="left 1 unmodeled row"): + builder._require_default_transfer_coverage(result, _frame()) + + +def test_group_quarters_rent_gap_is_explicit_structural_pending( + monkeypatch: pytest.MonkeyPatch, +) -> None: + builder = _load_builder_module() + frame = _frame(spines=("asec_puf", "acs_2024_1yr")) + frame.person["pre_subsidy_rent"] = [1_000.0, np.nan] + frame.table("household")["TYPEHUGQ"] = [np.nan, 3.0] + result = builder.AcsMultispineResult( + frame=frame, + provenance={ + "imputed_inputs": [ + { + "column": "pre_subsidy_rent", + "family": "housing", + "unmodeled_recipient_rows": 1, + } + ] + }, + ) + monkeypatch.setattr( + builder, + "default_acs_transfer_target_families", + lambda donor: {"person": {"housing": ("pre_subsidy_rent",)}}, + ) + + coverage = builder._require_default_transfer_coverage(result, _frame()) + + assert coverage["structural_pending"] == [ + { + "column": "pre_subsidy_rent", + "entity": "person", + "rows": 1, + "reason": ( + "ACS group-quarters rows are outside the housing-tenure universe" + ), + } + ] + + +def test_reviewed_limitations_close_gq_and_sub_puma_gaps() -> None: + builder = _load_builder_module() + frame = _frame(spines=("asec_puf", "acs_2024_1yr")) + frame.table("household")["TYPEHUGQ"] = [np.nan, 3.0] + result = builder.AcsMultispineResult( + frame=frame, + provenance={ + "geography_ladder": { + "applied": True, + "seed": 29, + "layer_vintages": { + "puma": "2020_puma", + "congressional_district": "119th_congress", + "county": "2020_census", + "tract": "2020_census", + }, + "unresolved_sub_puma_inputs": ["block_geoid", "tract_geoid"], + } + }, + ) + transfer_coverage = { + "structural_pending": [ + { + "column": "pre_subsidy_rent", + "entity": "person", + "rows": 1, + "reason": ( + "ACS group-quarters rows are outside the housing-tenure universe" + ), + } + ] + } + input_null_audit = [ + { + "entity": "person", + "column": "pre_subsidy_rent", + "missing_rows_by_spine": {"acs_2024_1yr": 1}, + }, + { + "entity": "person", + "column": "employment_income_before_lsr", + "missing_rows_by_spine": {"acs_2024_1yr": 1}, + }, + ] + + limitations = builder._reviewed_limitations( + result, + transfer_coverage=transfer_coverage, + input_null_audit=input_null_audit, + ) + + gq, native, geography = limitations + assert gq["status"] == "reviewed_structural_absence" + assert gq["affected_rows"] == {"household": 1, "person": 1, "spm_unit": 1} + assert gq["engine_input_nulls"] == [input_null_audit[0]] + assert gq["transfer_evidence"] == transfer_coverage["structural_pending"] + assert native["engine_input_nulls_excluding_group_quarters_housing"] == [ + input_null_audit[1] + ] + assert geography["status"] == "reviewed_probabilistic_assignment" + assert geography["unavailable_exact_geography"] == [ + "block_geoid", + "tract_geoid", + ] + assert geography["assignment_seed"] == 29 + assert all(item["calibration_blocker"] is False for item in limitations) + + +def test_donor_release_identity_verifies_sha_and_pins_provenance( + tmp_path: Path, +) -> None: + module = _load_builder_module() + sha = "a" * 64 + manifest = tmp_path / "release_manifest.json" + manifest.write_text( + json.dumps( + { + "dataset_role": "national_default", + "is_default": True, + "build": {"build_id": "populace-us-2024-buildo-sparse-x"}, + "artifacts": { + "populace_us_2024": { + "kind": "microdata", + "path": "populace_us_2024.h5", + "repo_id": "policyengine/populace-us", + "revision": "populace-us-2024-buildo-sparse-x", + "sha256": sha, + }, + "calibration_diagnostics": { + "kind": "diagnostics", + "path": "calibration_diagnostics.json", + "sha256": "b" * 64, + }, + }, + } + ) + ) + + identity = module._donor_release_identity(manifest, sha) + + assert identity == { + "manifest_path": str(manifest.resolve()), + "artifact": "populace_us_2024", + "release_id": "populace-us-2024-buildo-sparse-x", + "revision": "populace-us-2024-buildo-sparse-x", + "repo_id": "policyengine/populace-us", + "sha256": sha, + "dataset_role": "national_default", + "is_default": True, + } + assert module._donor_release_identity(None, sha) is None + + +def test_donor_release_identity_rejects_sha_mismatch(tmp_path: Path) -> None: + module = _load_builder_module() + manifest = tmp_path / "release_manifest.json" + manifest.write_text( + json.dumps( + { + "artifacts": { + "populace_us_2024": { + "kind": "microdata", + "sha256": "a" * 64, + } + } + } + ) + ) + + with pytest.raises(SystemExit, match="does not match its release manifest"): + module._donor_release_identity(manifest, "c" * 64) + + +def test_donor_release_identity_requires_one_microdata_artifact( + tmp_path: Path, +) -> None: + module = _load_builder_module() + manifest = tmp_path / "release_manifest.json" + manifest.write_text(json.dumps({"artifacts": {}})) + + with pytest.raises(SystemExit, match="exactly one microdata artifact"): + module._donor_release_identity(manifest, "a" * 64) diff --git a/tools/_legacy/build_us_acs_multispine_base.py b/tools/_legacy/build_us_acs_multispine_base.py new file mode 100644 index 00000000..7a5021a7 --- /dev/null +++ b/tools/_legacy/build_us_acs_multispine_base.py @@ -0,0 +1,1173 @@ +"""Build a nullable ASEC-by-PUF plus ACS multispine staging artifact. + +This tool starts from the dense, input-complete ASEC-by-PUF H5 produced by +the US input-family pipeline. It acquires the byte-pinned ACS PUMS archives, +loads and maps the ACS spine, transfers model input leaves from the dense +donor, assigns the PUMA-anchored state/CD/county geography ladder, audits every +fit's resolved typed weight kind, and writes the combined base. Calibration is +deliberately downstream. +""" + +from __future__ import annotations + +import argparse +import gc +import hashlib +import json +import os +import warnings +from dataclasses import replace +from pathlib import Path +from typing import Any + +import numpy as np +import pandas as pd + +from populace.build import ( + FitWeightRecord, + default_valued_columns_gate, + weights_audit_gate, +) +from populace.build.us_runtime import acs_sources +from populace.build.us_runtime.acs_multispine import ( + AcsMultispineResult, + build_optional_acs_multispine, +) +from populace.build.us_runtime.acs_pums import ( + ACS_2024_1YR_SPINE, + DEFAULT_CHUNKSIZE, + AcsPumsSource, +) +from populace.build.us_runtime.acs_transfer import ( + ACS_DONOR_CHANNEL_AUTO, + DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, + TargetFamilies, + acs_derived_transfer_expectations, + acs_transfer_donor_requirements, + declared_acs_transfer_target_families, + default_acs_transfer_target_families, + resolve_acs_donor_channel, +) +from populace.build.us_runtime.base_pool import spine_column +from populace.build.us_runtime.puma_ladder import ( + UsPumaLadder, + load_us_puma_ladder, +) +from populace.frame import Frame, WeightKind, Weights +from populace.frame.units import US_SCHEMA + +PERIOD = 2024 +# This implementation moved from tools/ into tools/_legacy/ unchanged. +_REPOSITORY_ROOT = Path(__file__).resolve().parents[2] +DEFAULT_INPUTS_DIR = _REPOSITORY_ROOT / "inputs" / "acs_2024_1yr" +DEFAULT_PUMA_LADDER = _REPOSITORY_ROOT / "build" / "us" / "us_puma_ladder_2020.npz" +DEFAULT_N_ESTIMATORS = 32 +DEFAULT_STAGING_EXPORT_PEAK_LIMIT_BYTES = int( + os.environ.get("POPULACE_STAGING_EXPORT_PEAK_LIMIT_BYTES", 30_000_000_000) +) +_STAGING_EXPORT_FIXED_OVERHEAD_BYTES = 512 * 1024**2 +_PACKAGED_MANIFEST_REFERENCE = ( + "package:populace.build.us_runtime/acs_2024_1yr_sources.json" +) + + +def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser( + description=( + "Append the pinned ACS 2024 1-year PUMS spine to an already-built " + "dense ASEC-by-PUF base. The nullable result has reviewed source-" + "universe limitations and is simulation-ready except for the " + "downstream calibration solve." + ) + ) + parser.add_argument("--base-h5", required=True, type=Path) + parser.add_argument("--out-h5", required=True, type=Path) + parser.add_argument( + "--summary", + type=Path, + help=( + "JSON build summary path. Defaults to OUT-H5 with a .summary.json suffix." + ), + ) + parser.add_argument( + "--source-manifest", + type=Path, + help="Strict ACS source manifest override; defaults to the packaged pin.", + ) + parser.add_argument( + "--inputs-dir", + default=DEFAULT_INPUTS_DIR, + type=Path, + help="Hash-verified ACS archive cache (default: inputs/acs_2024_1yr).", + ) + parser.add_argument( + "--puma-ladder", + default=DEFAULT_PUMA_LADDER, + type=Path, + help=( + "Validated national PUMA-ladder NPZ (default: " + "build/us/us_puma_ladder_2020.npz)." + ), + ) + parser.add_argument( + "--max-households", + type=_positive_int, + help="Optional deterministic smoke limit applied after ACS household sort.", + ) + parser.add_argument("--period", default=PERIOD, type=_positive_int) + parser.add_argument("--chunksize", default=DEFAULT_CHUNKSIZE, type=_positive_int) + parser.add_argument("--acs-share", default=0.5, type=_open_unit_interval) + parser.add_argument("--seed", default=0, type=_nonnegative_int) + parser.add_argument( + "--geography-seed", + default=0, + type=_nonnegative_int, + help="Deterministic PUMA/CD/county assignment seed (default: 0).", + ) + parser.add_argument( + "--n-estimators", + default=DEFAULT_N_ESTIMATORS, + type=_positive_int, + ) + parser.add_argument( + "--max-targets-per-fit", + default=DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, + type=_positive_int, + help="Maximum chained QRF targets retained at once (default: 8).", + ) + parser.add_argument( + "--donor-channel", + default=ACS_DONOR_CHANNEL_AUTO, + help=( + "ASEC-by-PUF support channel used by transfer. The default selects " + "the appropriate support channel for each input family." + ), + ) + parser.add_argument( + "--donor-release-manifest", + type=Path, + help=( + "release_manifest.json of the published release BASE-H5 came " + "from. When given, the donor H5's sha256 must match the " + "manifest's root microdata artifact, and the donor release " + "identity is pinned into the staging summary." + ), + ) + return parser.parse_args(argv) + + +def _donor_release_identity( + manifest_path: Path | None, + base_sha256: str, +) -> dict[str, object] | None: + """Verify the donor H5 against its release manifest and pin its identity.""" + + if manifest_path is None: + return None + try: + manifest = json.loads(manifest_path.read_text()) + except (OSError, ValueError) as exc: + raise SystemExit( + f"Cannot read donor release manifest {manifest_path}: {exc}." + ) from exc + artifacts = manifest.get("artifacts") + if not isinstance(artifacts, dict): + raise SystemExit( + f"Donor release manifest {manifest_path} carries no artifacts map." + ) + microdata = [ + (name, entry) + for name, entry in artifacts.items() + if isinstance(entry, dict) and entry.get("kind") == "microdata" + ] + if len(microdata) != 1: + raise SystemExit( + "Donor release manifest must carry exactly one microdata " + f"artifact; found {sorted(name for name, _ in microdata)}." + ) + name, entry = microdata[0] + if entry.get("sha256") != base_sha256: + raise SystemExit( + "Donor H5 sha256 does not match its release manifest: base-h5 " + f"is {base_sha256} but {manifest_path} pins " + f"{entry.get('sha256')!r} for {name!r}. Refusing to transfer " + "from an unverified donor." + ) + build = manifest.get("build") if isinstance(manifest.get("build"), dict) else {} + return { + "manifest_path": str(manifest_path.resolve()), + "artifact": name, + "release_id": build.get("build_id"), + "revision": entry.get("revision"), + "repo_id": entry.get("repo_id"), + "sha256": entry.get("sha256"), + "dataset_role": manifest.get("dataset_role"), + "is_default": manifest.get("is_default"), + } + + +def main(argv: list[str] | None = None) -> int: + args = _parse_args(argv) + summary_path = args.summary or args.out_h5.with_suffix(".summary.json") + _validate_artifact_paths(args, summary_path=summary_path) + + manifest = acs_sources.load_acs_source_manifest(args.source_manifest) + manifest_file = _manifest_file(args.source_manifest) + manifest_sha256 = _sha256(manifest_file) + base_sha256 = _sha256(args.base_h5) + donor_release = _donor_release_identity(args.donor_release_manifest, base_sha256) + base = _load_base_frame(args.base_h5) + _require_benefit_participation_inputs(base) + transfer_plan = declared_acs_transfer_target_families() + _require_dense_donor_coverage( + base, + donor_channel=args.donor_channel, + target_families=transfer_plan, + ) + base_rows = _row_counts(base) + base_mass = float(base.weights_for("household").total) + puma_ladder_sha256 = _sha256(args.puma_ladder) + puma_ladder = load_us_puma_ladder(args.puma_ladder) + + source = acs_sources.fetch_acs_pums_sources( + args.inputs_dir, + manifest=manifest, + ) + if args.max_households is not None: + source = replace(source, max_households=args.max_households) + + result = build_optional_acs_multispine( + base, + source, + chunksize=args.chunksize, + acs_share=args.acs_share, + target_families=transfer_plan, + donor_channel=args.donor_channel, + seed=args.seed, + n_estimators=args.n_estimators, + max_targets_per_fit=args.max_targets_per_fit, + puma_ladder=puma_ladder, + geography_seed=args.geography_seed, + ) + _require_puma_ladder_assignment(result) + _require_benefit_participation_transfer(result) + transfer_coverage = _require_default_transfer_coverage( + result, + base, + target_families=transfer_plan, + ) + weights_audit = _audit_fits(result) + + # The pooled frame owns its assembled blocks. Release the dense donor + # before HDF serialization so export cannot retain both full spines plus + # writer scratch at once. + del base + gc.collect() + input_null_audit = _engine_input_null_audit(result.frame) + gc.collect() + + args.out_h5.parent.mkdir(parents=True, exist_ok=True) + staging_export_peak_bytes = _preflight_staging_export(result.frame) + _write_dataset(result.frame, args.out_h5, period=args.period) + + summary = _build_summary( + args=args, + summary_path=summary_path, + manifest=manifest, + source=source, + base_rows=base_rows, + base_mass=base_mass, + base_sha256=base_sha256, + donor_release=donor_release, + manifest_sha256=manifest_sha256, + result=result, + weights_audit=weights_audit, + transfer_coverage=transfer_coverage, + input_null_audit=input_null_audit, + staging_export_peak_bytes=staging_export_peak_bytes, + puma_ladder=puma_ladder, + puma_ladder_sha256=puma_ladder_sha256, + ) + summary_path.parent.mkdir(parents=True, exist_ok=True) + rendered = json.dumps(summary, indent=2, sort_keys=True, allow_nan=False) + "\n" + summary_path.write_text(rendered, encoding="utf-8") + print(rendered, end="") + return 0 + + +def _audit_fits(result: AcsMultispineResult) -> dict[str, object]: + records = result.fit_records + if not records: + raise SystemExit( + "ACS input transfer produced no fit records. The dense donor must " + "carry transferable tax-detail and benefit-participation inputs." + ) + non_typed = [ + type(record).__name__ + for record in records + if not isinstance(record, FitWeightRecord) + ] + if non_typed: + raise TypeError( + "ACS input transfer returned non-FitWeightRecord audit evidence: " + f"{non_typed}." + ) + gate = weights_audit_gate(records) + if not gate.passed: + raise SystemExit("Weights audit failed:\n " + "\n ".join(gate.failures)) + return { + "passed": gate.passed, + "failures": list(gate.failures), + "details": dict(gate.details), + } + + +def _require_benefit_participation_inputs(base: Frame) -> None: + participation = sorted( + column + for entity in base.entities + for column in base.table(entity).columns + if column.startswith("takes_up_") + ) + if not participation: + raise SystemExit( + "Dense ASEC-by-PUF donor has no takes_up_* benefit-participation " + "inputs. This tool must run after the benefit input-family stages." + ) + + +def _require_dense_donor_coverage( + base: Frame, + engine: Any | None = None, + *, + donor_channel: str | None = ACS_DONOR_CHANNEL_AUTO, + target_families: TargetFamilies | None = None, +) -> None: + """Fail unless every column consumed by the QRF plan has donor signal.""" + + if engine is None: + from populace.frame.adapters.policyengine_us import PolicyEngineUSEngine + + engine = PolicyEngineUSEngine() + selected, resolved_channel = _coverage_donor_channel(base, donor_channel) + plan = target_families or declared_acs_transfer_target_families() + try: + requirements = acs_transfer_donor_requirements(selected, plan) + except ValueError as exc: + raise SystemExit( + "Dense ASEC-by-PUF donor has an invalid ACS transfer feature " + f"surface; selected channel={resolved_channel!r}: {exc}." + ) from exc + + failures: list[str] = [] + present_values: dict[str, Any] = {} + owners: dict[str, str] = {} + for entity, columns in requirements.items(): + if entity not in selected.entities: + failures.extend( + f"{entity}.{column}: transfer-consumed column is absent " + "because the donor entity is missing." + for column in columns + ) + continue + table = selected.table(entity) + for column in columns: + if column not in table.columns: + actual_owner = _column_owner(selected, column) + location = ( + f"; found on entity {actual_owner!r}" + if actual_owner is not None + else "" + ) + failures.append( + f"{entity}.{column}: transfer-consumed column is absent{location}." + ) + continue + values = table[column].to_numpy() + if not pd.notna(values).any(): + failures.append( + f"{entity}.{column}: transfer-consumed column has no " + "observed donor values." + ) + continue + present_values[column] = values + owners[column] = entity + + defaults = engine.default_values(sorted(present_values)) + default_gate = default_valued_columns_gate(present_values, defaults) + default_valued = default_gate.details["default_valued_columns"] + for column, default in sorted(default_valued.items()): + failures.append( + f"{owners[column]}.{column}: every observed donor value equals " + f"the engine default ({default!r}); QRF transfer requires usable " + "target/predictor signal." + ) + + if failures: + raise SystemExit( + "Dense ASEC-by-PUF donor failed the hard ACS transfer-consumption " + f"gate; selected channel={resolved_channel!r}:\n " + "\n ".join(failures) + ) + + +def _column_owner(frame: Frame, column: str) -> str | None: + try: + return frame.column_entity(column) + except ValueError: + return None + + +def _coverage_donor_channel( + base: Frame, + requested: str | None, +) -> tuple[Frame, str | None]: + try: + return resolve_acs_donor_channel(base, requested) + except ValueError as exc: + raise SystemExit(f"Invalid ACS transfer donor support metadata: {exc}") from exc + + +def _require_benefit_participation_transfer(result: AcsMultispineResult) -> None: + imputed = result.provenance.get("imputed_inputs", []) + transferred = isinstance(imputed, list) and any( + isinstance(item, dict) + and isinstance(item.get("family"), str) + and item["family"].split("__batch_", 1)[0] == "benefit_participation" + and isinstance(item.get("column"), str) + and item["column"].startswith("takes_up_") + for item in imputed + ) + if not transferred: + raise SystemExit( + "ACS default transfer produced no takes_up_* benefit-participation " + "input. Refusing to report a complete multispine base." + ) + + +def _require_puma_ladder_assignment(result: AcsMultispineResult) -> None: + """Fail unless the enabled multispine resolved launch geography inputs.""" + + raw = result.provenance.get("geography_ladder") + if not isinstance(raw, dict) or raw.get("applied") is not True: + raise SystemExit( + "ACS multispine did not apply the required PUMA geography ladder." + ) + household = result.frame.table("household") + required = ("puma", "congressional_district_geoid", "county_fips") + missing = [column for column in required if column not in household] + if missing: + raise SystemExit( + f"PUMA geography assignment omitted household column(s): {missing}." + ) + nulls = { + column: int(household[column].isna().sum()) + for column in required + if household[column].isna().any() + } + if nulls: + raise SystemExit( + f"PUMA geography assignment left null household values: {nulls}." + ) + deferred = result.provenance.get("deferred_inputs", []) + still_deferred = sorted( + {"congressional_district_geoid", "county_fips"}.intersection(deferred) + if isinstance(deferred, list) + else {"congressional_district_geoid", "county_fips"} + ) + if still_deferred: + raise SystemExit( + "PUMA geography assignment still reports resolved input(s) as " + f"deferred: {still_deferred}." + ) + + +def _require_default_transfer_coverage( + result: AcsMultispineResult, + donor: Frame, + *, + target_families: TargetFamilies | None = None, +) -> dict[str, object]: + """Prove every planned target is present and complete on its ACS universe.""" + + expected: dict[str, str] = {} + plan = target_families or default_acs_transfer_target_families(donor) + for entity, entity_families in plan.items(): + for targets in entity_families.values(): + for target in targets: + expected[target] = entity + # Deterministically derived columns are as load-bearing as fitted ones: + # a plan carrying the CGD parents owes the derived memo leg too. + expected.update(acs_derived_transfer_expectations(plan)) + raw_imputed = result.provenance.get("imputed_inputs", []) + if not isinstance(raw_imputed, list): + raise SystemExit("ACS imputed-input provenance must be a JSON list.") + entries = { + item["column"]: item + for item in raw_imputed + if isinstance(item, dict) and isinstance(item.get("column"), str) + } + if len(entries) != len( + [item for item in raw_imputed if isinstance(item, dict) and "column" in item] + ): + raise SystemExit("ACS imputed-input provenance contains duplicate columns.") + missing = sorted(set(expected) - set(entries)) + if missing: + raise SystemExit( + "ACS default transfer omitted donor-observed model input(s): " + f"{missing}. Refusing to report a complete multispine base." + ) + + structural_pending: list[dict[str, object]] = [] + for column, entity in sorted(expected.items()): + table = result.frame.table(entity) + if column not in table: + raise SystemExit( + f"ACS transfer registered {column!r} but the combined {entity!r} " + "table does not contain it." + ) + tag = spine_column(entity) + if tag not in table: + raise SystemExit( + f"Combined staging frame lacks ACS spine tag {tag!r} on {entity!r}." + ) + acs_mask = table[tag].eq(ACS_2024_1YR_SPINE) + if not acs_mask.any(): + raise SystemExit(f"Combined staging frame has no ACS rows on {entity!r}.") + missing_mask = table[column].isna() & acs_mask + missing_rows = int(missing_mask.sum()) + raw_unmodeled = entries[column].get("unmodeled_recipient_rows", 0) + if type(raw_unmodeled) is not int or raw_unmodeled < 0: + raise SystemExit( + f"ACS imputation provenance for {column!r} has invalid " + f"unmodeled_recipient_rows={raw_unmodeled!r}." + ) + if missing_rows != raw_unmodeled: + raise SystemExit( + f"ACS transfer provenance for {column!r} reports " + f"{raw_unmodeled} unmodeled row(s), but the combined ACS spine " + f"contains {missing_rows} missing row(s)." + ) + if missing_rows == 0: + continue + if column != "pre_subsidy_rent" or entity != "person": + raise SystemExit( + f"ACS default transfer left {missing_rows} unmodeled row(s) in " + f"required input {column!r}." + ) + gq_mask = _acs_group_quarters_person_mask(result.frame) + if not missing_mask.equals(gq_mask): + raise SystemExit( + "ACS pre_subsidy_rent may remain absent only for native " + "group-quarters rows." + ) + structural_pending.append( + { + "column": column, + "entity": entity, + "rows": missing_rows, + "reason": "ACS group-quarters rows are outside the housing-tenure universe", + } + ) + + return { + "expected_inputs": sorted(expected), + "registered_inputs": sorted(entries), + "structural_pending": structural_pending, + } + + +def _acs_group_quarters_person_mask(frame: Frame) -> pd.Series: + household = frame.table("household") + person = frame.table("person") + required = {"household_id", "TYPEHUGQ", spine_column("household")} + missing = sorted(required - set(household.columns)) + if missing: + raise SystemExit( + "Cannot validate ACS group-quarters transfer gaps; household table " + f"lacks {missing}." + ) + kind = pd.to_numeric(household["TYPEHUGQ"], errors="coerce") + gq_households = household.loc[ + household[spine_column("household")].eq(ACS_2024_1YR_SPINE) & kind.isin([2, 3]), + "household_id", + ] + return person[spine_column("person")].eq(ACS_2024_1YR_SPINE) & person[ + "person_household_id" + ].isin(gq_households) + + +def _reviewed_limitations( + result: AcsMultispineResult, + *, + transfer_coverage: dict[str, object], + input_null_audit: list[dict[str, object]], +) -> list[dict[str, object]]: + """Document accepted source-universe and geographic precision limits.""" + + geography = result.provenance["geography_ladder"] + if not isinstance(geography, dict): # guarded by _require_puma_ladder_assignment + raise TypeError("geography_ladder provenance must be a mapping.") + + acs_nulls = [ + entry + for entry in input_null_audit + if isinstance(entry.get("missing_rows_by_spine"), dict) + and int(entry["missing_rows_by_spine"].get(ACS_2024_1YR_SPINE, 0)) > 0 + ] + gq_counts = _acs_group_quarters_counts(result.frame) + gq_entity_by_column = { + "pre_subsidy_rent": "person", + "spm_unit_tenure_type": "spm_unit", + "tenure_type": "household", + } + + def is_exact_gq_null(entry: dict[str, object]) -> bool: + column = entry.get("column") + entity = gq_entity_by_column.get(column) + missing_by_spine = entry.get("missing_rows_by_spine") + return ( + entity is not None + and entry.get("entity") == entity + and isinstance(missing_by_spine, dict) + and int(missing_by_spine.get(ACS_2024_1YR_SPINE, 0)) == gq_counts[entity] + ) + + gq_nulls = [entry for entry in acs_nulls if is_exact_gq_null(entry)] + other_native_nulls = [entry for entry in acs_nulls if not is_exact_gq_null(entry)] + structural_pending = transfer_coverage.get("structural_pending", []) + if not isinstance(structural_pending, list): + raise TypeError("transfer_coverage.structural_pending must be a list.") + + return [ + { + "id": "acs_group_quarters_housing_universe", + "status": "reviewed_structural_absence", + "affected_spine": ACS_2024_1YR_SPINE, + "affected_rows": gq_counts, + "affected_columns": { + "household": [ + "tenure_type", + "acs_monthly_contract_rent", + "acs_monthly_gross_rent", + "acs_annual_property_tax", + ], + "spm_unit": ["spm_unit_tenure_type"], + "person": ["pre_subsidy_rent", "real_estate_taxes"], + }, + "reason": ( + "ACS PUMS housing-unit tenure, rent, and property-tax fields " + "are outside the TYPEHUGQ 2/3 group-quarters universe." + ), + "treatment": ( + "Preserve those values as structural nulls; filling them with " + "zero or donor housing values would synthesize an unobserved " + "housing unit." + ), + "engine_input_nulls": gq_nulls, + "transfer_evidence": structural_pending, + "calibration_blocker": False, + }, + { + "id": "native_acs_source_universe_blanks", + "status": "reviewed_source_missingness", + "affected_spine": ACS_2024_1YR_SPINE, + "reason": ( + "Native ACS inputs retain official blank universes; the ACS " + "mapping contract forbids inventing zeros or component splits." + ), + "treatment": ( + "Preserve nullable source semantics after transferring every " + "donor-observed model-required input with an eligible fit." + ), + "engine_input_nulls_excluding_group_quarters_housing": (other_native_nulls), + "calibration_blocker": False, + }, + { + "id": "sub_puma_geographic_precision", + "status": "reviewed_probabilistic_assignment", + "observed_geography": { + "acs_2024_1yr": ["state_fips", "puma"], + "asec_puf": ( + [ + "state_fips", + "tract_geoid", + "congressional_district_geoid", + "county_fips", + ] + if geography.get("donor_geography") == "preserved_assigned" + else ["state_fips"] + ), + }, + "assigned_geography": [ + "puma", + "congressional_district_geoid", + "county_fips", + ], + "donor_geography": geography.get("donor_geography"), + "unavailable_exact_geography": list( + geography.get( + "unresolved_sub_puma_inputs", + ["block_geoid", "tract_geoid"], + ) + ), + "unavailable_exact_geography_scope": ( + "acs_2024_1yr" + if geography.get("donor_geography") == "preserved_assigned" + else "acs_2024_1yr,asec_puf" + ), + "reason": ( + "ACS PUMS identifies residence only through state and 2020 " + "PUMA; exact block, tract, county, and congressional district " + "cannot be recovered for a source microrecord." + ), + "treatment": ( + ( + "Retain each ACS record's observed PUMA and assign its " + "119th-CD/county from official population-weighted PUMA " + "overlaps using the recorded seed. Donor records keep " + "their certified block-ladder district/county, with PUMA " + "derived exactly from the assigned 2020 tract. Do not " + "synthesize ACS block or tract." + ) + if geography.get("donor_geography") == "preserved_assigned" + else ( + "Retain each ACS record's observed PUMA, draw ASEC PUMA " + "within native state, and assign 119th-CD/county from " + "official population-weighted PUMA overlaps using the " + "recorded seed. Do not synthesize block or tract." + ) + ), + "assignment_seed": geography.get("seed"), + "layer_vintages": geography.get("layer_vintages", {}), + "calibration_blocker": False, + }, + ] + + +def _acs_group_quarters_counts(frame: Frame) -> dict[str, int]: + household = frame.table("household") + person = frame.table("person") + household_tag = spine_column("household") + required_household = {"household_id", "TYPEHUGQ", household_tag} + required_person = { + "person_household_id", + "person_spm_unit_id", + spine_column("person"), + } + missing = sorted( + (required_household - set(household.columns)) + | (required_person - set(person.columns)) + ) + if missing: + raise SystemExit( + "Cannot summarize reviewed ACS group-quarters limitations; " + f"combined frame lacks {missing}." + ) + kind = pd.to_numeric(household["TYPEHUGQ"], errors="coerce") + household_mask = household[household_tag].eq(ACS_2024_1YR_SPINE) & kind.isin([2, 3]) + household_ids = household.loc[household_mask, "household_id"] + person_mask = person[spine_column("person")].eq(ACS_2024_1YR_SPINE) & person[ + "person_household_id" + ].isin(household_ids) + return { + "household": int(household_mask.sum()), + "person": int(person_mask.sum()), + "spm_unit": int(person.loc[person_mask, "person_spm_unit_id"].nunique()), + } + + +def _build_summary( + *, + args: argparse.Namespace, + summary_path: Path, + manifest: acs_sources.AcsSourceManifest, + source: AcsPumsSource, + base_rows: dict[str, int], + base_mass: float, + base_sha256: str, + donor_release: dict[str, object] | None, + manifest_sha256: str, + result: AcsMultispineResult, + weights_audit: dict[str, object], + transfer_coverage: dict[str, object], + input_null_audit: list[dict[str, object]], + staging_export_peak_bytes: int, + puma_ladder: UsPumaLadder, + puma_ladder_sha256: str, +) -> dict[str, object]: + output_rows = _row_counts(result.frame) + output_mass = float(result.frame.weights_for("household").total) + return { + "version": 1, + "stage": "acs_2024_1yr_multispine_base", + "artifact_kind": "nullable_precalibration_staging_h5", + "calibration_applied": False, + "simulation_ready": False, + "simulation_ready_except_calibration": True, + "simulation_readiness_blockers": ["calibration_not_applied"], + "reviewed_limitations": _reviewed_limitations( + result, + transfer_coverage=transfer_coverage, + input_null_audit=input_null_audit, + ), + "period": args.period, + "base": { + "path": str(args.base_h5.resolve()), + "sha256": base_sha256, + "rows": base_rows, + "household_weight_total": base_mass, + "donor_release": donor_release, + }, + "acs_sources": _source_provenance( + manifest, + source, + manifest_path=args.source_manifest, + manifest_sha256=manifest_sha256, + ), + "geography_ladder": { + "path": str(args.puma_ladder.resolve()), + "sha256": puma_ladder_sha256, + "pumas": len(puma_ladder), + "layer_vintages": puma_ladder.layer_vintages, + "seed": args.geography_seed, + "assignment": result.provenance["geography_ladder"], + }, + "orchestration": { + "chunksize": args.chunksize, + "acs_share": args.acs_share, + "max_households": args.max_households, + "seed": args.seed, + "geography_seed": args.geography_seed, + "n_estimators": args.n_estimators, + "max_targets_per_fit": args.max_targets_per_fit, + "donor_channel": args.donor_channel, + "provenance": result.provenance, + }, + "weights_audit": weights_audit, + "transfer_coverage": transfer_coverage, + "reviewed_engine_input_nulls": input_null_audit, + "staging_export_peak_estimate_bytes": staging_export_peak_bytes, + "rows": { + "base": base_rows, + "combined": output_rows, + }, + "household_weight_totals": { + "base": base_mass, + "combined": output_mass, + }, + "spine_totals": _spine_totals(result.frame), + "output": { + "path": str(args.out_h5.resolve()), + "sha256": _sha256(args.out_h5), + "summary_path": str(summary_path.resolve()), + "rows": output_rows, + "household_weight_total": output_mass, + }, + } + + +def _source_provenance( + manifest: acs_sources.AcsSourceManifest, + source: AcsPumsSource, + *, + manifest_path: Path | None, + manifest_sha256: str, +) -> dict[str, object]: + resolved_manifest_path = _manifest_file(manifest_path) + local_paths = { + "household": source.household_zip.resolve(), + "person": source.person_zip.resolve(), + } + return { + "manifest": ( + str(resolved_manifest_path) + if manifest_path is not None + else _PACKAGED_MANIFEST_REFERENCE + ), + "manifest_sha256": manifest_sha256, + "version": manifest.version, + "spine": manifest.spine, + "vintage": manifest.vintage, + "verified_on": manifest.verified_on, + "source_directory": manifest.source_directory, + "artifacts": [ + { + "role": artifact.role, + "filename": artifact.filename, + "url": artifact.url, + "sha256": artifact.sha256, + "size_bytes": artifact.size_bytes, + "local_path": str(local_paths[artifact.role]), + } + for artifact in manifest.artifacts + ], + } + + +def _manifest_file(override: Path | None) -> Path: + if override is not None: + return override.resolve() + return Path(acs_sources.__file__).with_name("acs_2024_1yr_sources.json") + + +def _validate_artifact_paths( + args: argparse.Namespace, + *, + summary_path: Path, +) -> None: + base = args.base_h5.resolve() + output = args.out_h5.resolve() + summary = summary_path.resolve() + if args.out_h5.suffix != ".h5": + raise SystemExit(f"--out-h5 must end with .h5, got {args.out_h5.name!r}.") + if base == output: + raise SystemExit("--out-h5 must differ from --base-h5.") + if summary in {base, output}: + raise SystemExit("--summary must differ from both --base-h5 and --out-h5.") + ladder = args.puma_ladder.resolve() + if ladder == output: + raise SystemExit("--puma-ladder must differ from --out-h5.") + + +def _spine_totals(frame: Frame) -> dict[str, dict[str, Any]]: + spine_values: set[str] = set() + for entity in frame.entities: + column = spine_column(entity) + table = frame.table(entity) + if column not in table: + raise ValueError(f"Combined ACS base lacks required spine tag {column!r}.") + if table[column].isna().any(): + raise ValueError( + f"Combined ACS base carries missing values in spine tag {column!r}." + ) + spine_values.update(map(str, table[column].dropna().unique())) + + household = frame.table("household") + household_spine = household[spine_column("household")] + weights = pd.Series( + frame.weights_for("household").values, + index=household.index, + ) + return { + spine: { + "rows": { + entity: int(frame.table(entity)[spine_column(entity)].eq(spine).sum()) + for entity in frame.entities + }, + "household_weight_total": float( + weights.loc[household_spine.eq(spine)].sum() + ), + } + for spine in sorted(spine_values) + } + + +def _load_base_frame(path: Path) -> Frame: + """Load the dense donor H5 without importing PolicyEngine-US at tool import.""" + + from policyengine_us.data import USSingleYearDataset + + dataset = USSingleYearDataset(file_path=str(path)) + tables = { + "person": dataset.person, + "household": dataset.household, + "tax_unit": dataset.tax_unit, + "spm_unit": dataset.spm_unit, + "family": dataset.family, + "marital_unit": dataset.marital_unit, + } + household_weights = ( + tables["household"].pop("household_weight").to_numpy(dtype=np.float64) + ) + return Frame( + tables, + US_SCHEMA, + { + "household": Weights( + household_weights, + WeightKind.CALIBRATED, + ) + }, + ) + + +def _write_dataset( + frame: Frame, + path: Path, + *, + period: int, + artifact_kind: str = "nullable_precalibration_staging_h5", +) -> None: + """Write a populace US H5 with one-table-at-a-time verification.""" + + output = Path(path) + output.unlink(missing_ok=True) + try: + with pd.HDFStore(output, mode="w") as store: + for entity in frame.entities: + table = frame.table(entity) + if entity == "household": + table = table.copy() + table["household_weight"] = frame.weights_for("household").values + if len(table): + # Fixed format preserves mixed bool/null object columns + # losslessly. Table format rejects them, which would force + # an unauthorized fill or type rewrite on base-only inputs. + with warnings.catch_warnings(): + warnings.simplefilter("ignore", pd.errors.PerformanceWarning) + store.put(entity, table, format="fixed") + store.put( + "_time_period", + pd.Series([int(period)]), + format="table", + ) + store.put( + "_populace_staging_metadata", + pd.Series( + [ + json.dumps( + { + "artifact_kind": artifact_kind, + "entity_hdf_format": "fixed_nullable", + "household_weight_kind": frame.weights_for( + "household" + ).kind.value, + }, + sort_keys=True, + ) + ] + ), + format="table", + ) + + with pd.HDFStore(output, mode="r") as store: + for entity in frame.entities: + expected = frame.table(entity) + if not len(expected): + continue + stored = store[entity] + expected_columns = list(expected.columns) + if entity == "household": + expected_columns.append("household_weight") + if ( + len(stored) != len(expected) + or list(stored.columns) != expected_columns + ): + raise RuntimeError( + f"Staging H5 round trip changed {entity!r}: expected " + f"{len(expected)} rows/{expected_columns}, got " + f"{len(stored)} rows/{list(stored.columns)}." + ) + del stored + except BaseException: + output.unlink(missing_ok=True) + raise + + +def _engine_input_null_audit( + frame: Frame, + engine: Any | None = None, +) -> list[dict[str, object]]: + """Inventory nullable engine inputs for the reviewed-limitations summary.""" + + if engine is None: + from populace.frame.adapters.policyengine_us import PolicyEngineUSEngine + + engine = PolicyEngineUSEngine() + input_names = set(engine.variables()) + entries: list[dict[str, object]] = [] + for entity in frame.entities: + table = frame.table(entity) + tag = spine_column(entity) + for column in sorted(set(table.columns).intersection(input_names)): + missing = table[column].isna() + if not missing.any(): + continue + by_spine: dict[str, int] = {} + if tag in table: + by_spine = { + str(spine): int((missing & table[tag].eq(spine)).sum()) + for spine in sorted(map(str, table[tag].dropna().unique())) + if int((missing & table[tag].eq(spine)).sum()) + } + entries.append( + { + "entity": entity, + "column": column, + "dtype": engine.variable_metadata(column).dtype, + "missing_rows": int(missing.sum()), + "rows": len(table), + "missing_rows_by_spine": by_spine, + } + ) + return entries + + +def _preflight_staging_export( + frame: Frame, + *, + max_peak_bytes: int = DEFAULT_STAGING_EXPORT_PEAK_LIMIT_BYTES, +) -> int: + if type(max_peak_bytes) is not int or max_peak_bytes <= 0: + raise ValueError("max_peak_bytes must be a positive integer.") + table_bytes = { + entity: int(frame.table(entity).memory_usage(index=True, deep=True).sum()) + for entity in frame.entities + } + resident = sum(table_bytes.values()) + resident += sum( + frame.weights_for(entity).values.nbytes for entity in frame.weighted_entities + ) + resident += int(frame.strata.memory_usage(index=True, deep=True)) + largest_table = max(table_bytes.values(), default=0) + household_copy = table_bytes.get("household", 0) + 8 * frame.n("household") + estimate = int( + resident + + 2 * largest_table + + household_copy + + _STAGING_EXPORT_FIXED_OVERHEAD_BYTES + ) + if estimate > max_peak_bytes: + raise MemoryError( + "Nullable ACS staging export is estimated to require " + f"{estimate / 1_000_000_000:.2f} GB, above the " + f"{max_peak_bytes / 1_000_000_000:.2f} GB limit." + ) + return estimate + + +def _row_counts(frame: Frame) -> dict[str, int]: + return {entity: frame.n(entity) for entity in frame.entities} + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as file: + while chunk := file.read(1024 * 1024): + digest.update(chunk) + return digest.hexdigest() + + +def _positive_int(value: str) -> int: + parsed = int(value) + if parsed <= 0: + raise argparse.ArgumentTypeError("must be a positive integer") + return parsed + + +def _nonnegative_int(value: str) -> int: + parsed = int(value) + if parsed < 0: + raise argparse.ArgumentTypeError("must be a non-negative integer") + return parsed + + +def _open_unit_interval(value: str) -> float: + parsed = float(value) + if not 0.0 < parsed < 1.0: + raise argparse.ArgumentTypeError("must be strictly between 0 and 1") + return parsed + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/build_us_acs_local_release.py b/tools/build_us_acs_local_release.py index 2a5ec9d9..c65adb5f 100644 --- a/tools/build_us_acs_local_release.py +++ b/tools/build_us_acs_local_release.py @@ -74,6 +74,14 @@ ARTIFACT_NAME = "populace_us_2024_acs_local" ARTIFACT_FILENAME = f"{ARTIFACT_NAME}.h5" HF_REPO_ID = "policyengine/populace-us" +LEGACY_STAGING_REFRESH_RECIPE = ( + "uv run tools/build_us_acs_multispine_base.py " + "--base-h5 .h5 " + "--donor-release-manifest " + "--out-h5 /acs_multispine_staging.h5 " + "--inputs-dir " + "--puma-ladder build/us/us_puma_ladder_2020.npz" +) # --------------------------------------------------------------------------- @@ -1658,14 +1666,7 @@ def _version(package: str) -> str: "against the new certified release H5; everything else is " "unchanged." ), - "staging": ( - "uv run tools/build_us_acs_multispine_base.py " - "--base-h5 .h5 " - "--donor-release-manifest " - "--out-h5 /acs_multispine_staging.h5 " - "--inputs-dir " - "--puma-ladder build/us/us_puma_ladder_2020.npz" - ), + "staging": LEGACY_STAGING_REFRESH_RECIPE, "release": ( "uv run tools/build_us_acs_local_release.py --stage all " "--staging-h5 /acs_multispine_staging.h5 " diff --git a/tools/build_us_acs_multispine_base.py b/tools/build_us_acs_multispine_base.py index e3529e26..dfbe9a36 100644 --- a/tools/build_us_acs_multispine_base.py +++ b/tools/build_us_acs_multispine_base.py @@ -1,15 +1,18 @@ -"""Deprecated compatibility shim for the retired late-ACS builder. +"""Deprecated entry point for the preserved ACS local-release staging builder. -The executable build path moved to ``tools/build_us_multispine_pool.py``. -This module keeps only the two H5 helpers imported by the legacy ACS -local-release tool. It never runs or translates the retired late-assembly -pipeline. +The late-ACS staging lineage remains supported until populace#578 increment 4 +retires the local-release overlay. Its implementation lives under +``tools/_legacy`` so new work cannot mistake it for the assembly-first pool +builder, while this historical command and its helper imports keep working. """ from __future__ import annotations +import importlib.util import sys +import warnings from pathlib import Path +from types import ModuleType from populace.build.us_runtime.h5_io import ( LEGACY_NULLABLE_STAGING_ARTIFACT_KIND, @@ -20,19 +23,44 @@ __all__ = ["_load_base_frame", "_write_dataset", "main"] -_MIGRATION_MESSAGE = """\ -tools/build_us_acs_multispine_base.py is retired. +_LEGACY_PATH = ( + Path(__file__).resolve().parent + / "_legacy" + / "build_us_acs_multispine_base.py" +) +_DEPRECATION_MESSAGE = ( + "tools/build_us_acs_multispine_base.py is deprecated and remains available " + "only for the supported ACS local-release chain. It will be removed by " + "populace#578 increment 4; new multispine builds must use " + "tools/build_us_multispine_pool.py." +) -Its --base-h5 input was already post-clone, so the legacy command cannot be translated -without violating the required assemble -> clone ordering. -Use the sha-pinned pool builder instead: - uv run tools/build_us_multispine_pool.py --help -""" +def _load_legacy_module() -> ModuleType: + """Load the moved implementation without making ``tools`` a package.""" + + module_name = "_populace_legacy_us_acs_multispine_base" + existing = sys.modules.get(module_name) + if existing is not None: + return existing + spec = importlib.util.spec_from_file_location(module_name, _LEGACY_PATH) + if spec is None or spec.loader is None: + raise ImportError(f"Cannot load preserved legacy builder at {_LEGACY_PATH}.") + module = importlib.util.module_from_spec(spec) + sys.modules[module_name] = module + try: + spec.loader.exec_module(module) + except BaseException: + sys.modules.pop(module_name, None) + raise + return module + + +_legacy = _load_legacy_module() def _load_base_frame(path: Path) -> Frame: - """Load a legacy calibrated US H5 for the deprecated local-release lane.""" + """Load a legacy calibrated US H5 for the local-release lane.""" return load_legacy_calibrated_us_h5(path) @@ -44,7 +72,7 @@ def _write_dataset( period: int, artifact_kind: str = LEGACY_NULLABLE_STAGING_ARTIFACT_KIND, ) -> None: - """Write the legacy lane's nullable H5 through the shared atomic writer.""" + """Write a legacy-lane H5 through the shared verified atomic writer.""" write_nullable_us_h5( frame, @@ -55,11 +83,10 @@ def _write_dataset( def main(argv: list[str] | None = None) -> int: - """Refuse the retired CLI and name the explicit migration command.""" + """Warn, then run the preserved local-release staging implementation.""" - del argv - print(_MIGRATION_MESSAGE, file=sys.stderr) - return 2 + warnings.warn(_DEPRECATION_MESSAGE, DeprecationWarning, stacklevel=2) + return _legacy.main(argv) if __name__ == "__main__": From cb5f86366efc6c8116a907779e079eb5155aacaa Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 21:52:47 -0400 Subject: [PATCH 28/94] Checkpoint legacy release compatibility --- PROGRESS.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 8126c69f..7fd2d1e4 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -25,15 +25,19 @@ the deprecated-but-supported local ACS release producer. under one run ID, publish them atomically, and write the readiness manifest last in `b75f352`. The three reviewer interruption points now retain a non-ready tombstone; focused H5/pool coverage passes (17 tests). +- Restored the deprecated ACS staging CLI by delegating to the preserved + pre-shim implementation in `tools/_legacy` in `f8bcb3b`. Its local-release + recipe, summary/reviewed-null contract, helper compatibility, and legacy + suite pass (37 tests); focused Ruff is clean. - Established the required final constraint: this live checkpoint will be restored byte-for-byte to `origin/main` before handoff so no root journal ships in the PR diff. ## Next -- Finish tracing the producer/pool and legacy-release artifact contracts. +- Finish tracing the producer/pool raw artifact and operator contracts. - Add each reviewer repro as a failing regression before its implementation fix. -- Fix and commit the raw-boundary and legacy-shim blockers. +- Fix and commit the raw-boundary blocker. - Run the full `populace-build` suite and Ruff, write the external handoff, and restore `PROGRESS.md` to `origin/main`. From 5b930ab43dfb6bb1550152abfb5a6584d68aa7d4 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 22:50:46 -0400 Subject: [PATCH 29/94] Make multispine lineage assembly-first --- PROGRESS.md | 26 +- .../578-multispine-pool-build.added.md | 11 +- docs/us-multispine-operator-ordering.md | 105 ++- .../src/populace/build/us_runtime/__init__.py | 20 + .../build/us_runtime/asec_checkpoint.py | 342 ++++++++- .../src/populace/build/us_runtime/h5_io.py | 223 ++++++ .../build/us_runtime/multispine_pool.py | 672 +++++++++++++++++- .../build/us_runtime/operator_boundary.py | 471 ++++++++++++ .../build/us_runtime/spine_agreement.py | 300 +++++++- .../tests/test_us_asec_checkpoint.py | 324 ++++++++- .../tests/test_us_multispine_pool.py | 350 ++++++++- .../tests/test_us_multispine_pool_tool.py | 213 +++++- .../tests/test_us_puf_support_base_builder.py | 320 +++++++++ .../tests/test_us_spine_agreement.py | 75 ++ .../tests/test_us_spine_blindness.py | 1 + tools/build_us_multispine_pool.py | 127 +++- tools/build_us_puf_support_base.py | 197 ++++- 17 files changed, 3670 insertions(+), 107 deletions(-) create mode 100644 packages/populace-build/src/populace/build/us_runtime/operator_boundary.py diff --git a/PROGRESS.md b/PROGRESS.md index 7fd2d1e4..cfa9284b 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -29,15 +29,29 @@ the deprecated-but-supported local ACS release producer. pre-shim implementation in `tools/_legacy` in `f8bcb3b`. Its local-release recipe, summary/reviewed-null contract, helper compatibility, and legacy suite pass (37 tests); focused Ruff is clean. +- Split the pooled ASEC producer at a dedicated, operator-untouched raw-stage + artifact; its exact `LKWEEKS`/`ED_VAL` mappings, artifact binding, resume + repair, complete operator-family exclusion, and unchanged legacy checkpoint + sequence are covered by regressions. +- Rewired the multispine pool to consume only that raw artifact, assemble and + clone first, run the full 20-operator source-input chain on raw-evidenced + rows, preserve ACS-native cells, and transfer the remaining nullable peer + inputs with explicit ownership. Consolidated blocker coverage passes, + including the legacy local-release path and structural guard. +- Closed the independent lineage-review findings: ACS native exceptions bind + exact mapping contracts, source-kernel projections carry no false full-pool + receipt, production wiring and the immutable 20-operator order are tested, + and the pool-specific agreement registry covers every expanded transfer, + take-up, SSI, and joint immigration surface. +- Added a readiness loader that accepts only a green manifest whose publication + run ID and digests match the H5 metadata and agreement diagnostics; the three + interruption regressions now exercise that reader as well as the tombstone. - Established the required final constraint: this live checkpoint will be restored byte-for-byte to `origin/main` before handoff so no root journal ships in the PR diff. ## Next -- Finish tracing the producer/pool raw artifact and operator contracts. -- Add each reviewer repro as a failing regression before its implementation - fix. -- Fix and commit the raw-boundary blocker. -- Run the full `populace-build` suite and Ruff, write the external handoff, and - restore `PROGRESS.md` to `origin/main`. +- Commit the reviewed raw-boundary and readiness integration. +- Run the full `populace-build` suite and Ruff. +- Write the external handoff and restore `PROGRESS.md` to `origin/main`. diff --git a/changelog.d/578-multispine-pool-build.added.md b/changelog.d/578-multispine-pool-build.added.md index 64e84e30..24bebd34 100644 --- a/changelog.d/578-multispine-pool-build.added.md +++ b/changelog.d/578-multispine-pool-build.added.md @@ -1 +1,10 @@ -Add a SHA-pinned, pre-calibration US multispine pool builder that assembles ASEC and ACS before PUF cloning and source-blind operators, emits nullable input-only artifacts with provenance receipts, and runs the fixed spine-agreement gate as its terminal stage; this code increment has synthetic fixture coverage but does not certify a full-data build. +Add a SHA-pinned, pre-calibration US multispine pool builder that consumes a +dedicated operator-untouched ASEC raw-stage artifact, assembles ASEC and ACS +before PUF cloning and the full source-blind operator chain, publishes nullable +input-only artifacts through a run-bound invalidate-then-publish protocol, and +runs the fixed spine-agreement gate over the complete pool transfer charter, +including categorical joint checks, as its terminal stage; the readiness +loader verifies the manifest, H5, and diagnostics as one publication, the +deprecated ACS local-release builder remains executable until increment 4, +and this code increment has synthetic fixture coverage but does not certify a +full-data build. diff --git a/docs/us-multispine-operator-ordering.md b/docs/us-multispine-operator-ordering.md index 0e0742c5..fc8bf644 100644 --- a/docs/us-multispine-operator-ordering.md +++ b/docs/us-multispine-operator-ordering.md @@ -60,53 +60,105 @@ transfers from a donor after the donor has crossed the ASEC-only operator sequence. Appending the transferred ACS records later does not cause those operators to run over the combined population. -`build_us_acs_multispine_base.py` is now a deprecated compatibility shim for -shared legacy H5 helpers. It does not expose a second executable pool builder. +`build_us_acs_multispine_base.py` remains a deprecated but executable +compatibility path until populace#578 increment 4 retires the ACS local-release +overlay. The public command warns and delegates to the preserved implementation +under `tools/_legacy`; its summary and reviewed-null receipts remain the inputs +expected by `build_us_acs_local_release.py`. New multispine work uses the pool +builder below, but the supported legacy release recipe is not left half-working. ## Executable increment-2 pool build `build_us_multispine_pool.py` consumes only explicit local files and their declared SHA-256 values: -- the input-complete ASEC checkpoint from the - `pre_clone_enrichment` outer-stage boundary; +- the dedicated `populace_us_asec_raw_stage` artifact emitted alongside the + producer's `source_construction` checkpoint. Its stage tag is + `raw_source_mapping` and its operator status is `operator_untouched`; - the ACS household and person PUMS archives, whose caller-supplied hashes - must also match the checked-in ACS source manifest; and + must also match the checked-in ACS source manifest; +- the canonical ACS 2022 rent donor used by the post-assembly housing + operator; and - the processed PUF H5 and source-year PUF CSV used by the existing donor loader. -The tool does not download any source. It verifies all file pins before -loading frames, maps measured ACS fields without overwriting them, and then -runs this fixed sequence: +The raw artifact is a second producer output, not a relabeling of +`pre_clone_enrichment`. It contains pooled ASEC unit structure and measured raw +columns. The only enrichment allowed there is faithful source mapping: +`LKWEEKS` and `ED_VAL` are restored by exact, pinned Census identity joins. +No `weeks_unemployed`, `educational_assistance`, carried-income split, +eligibility, pregnancy, take-up, childcare, retirement, or immigration output +is present. The producer still emits its historical enriched checkpoint and +final H5 for the sparse/dense single-spine release lineage; their operator +sequence and bytes are unchanged. + +The dedicated raw artifact is produced by the checkpointed producer recipe +(`--stage all --checkpoint-dir ...`) at +`/asec_raw_stage.checkpoint.h5`. The legacy monolithic recipe +continues to produce only its historical release outputs. + +The tool does not download any source. It verifies all file pins and validates +the raw artifact kind, stage, frame identity, operator status, and complete +operator-output absence before loading the peer frames. Measured ACS mappings +are allowed only when named by the ACS native-input receipt. It then runs this +fixed sequence: 1. `assemble_spines({"asec": ..., "acs": ...})` creates the first shared population state and binds the immutable assembly receipt. 2. `clone_us_frame_for_puf_support(...)` applies the PUF-detail clone to the whole assembled pool. Clone-index provenance, not source-spine identity, controls later PUF-detail routing. -3. The primary PUF QRF chain and capital-gains tail transfer run over the - combined frame, followed by the declared ACS input-family QRF transfers. - Existing measured target cells remain unchanged, and transfer receipts - record fitted and imputed rows. -4. Deterministic input reconciliation runs over that same pool. -5. The seed stage preserves existing take-up values, applies the sourced +3. CPS-carried predictor inputs are derived after assembly on rows with the + required measured CPS source fields. The pool wrapper validates the complete + assembly receipt first, then gives the historical CPS kernel an ephemeral + `PERIDNUM`-available structural projection. That projection deliberately + carries neither the full-pool assembly receipt nor its mass history. Only + the kernel's declared output family is merged back into the still-receipted + full pool. Unavailable peer rows remain nullable; no operator uses + source-channel identity to choose behavior. +4. The primary PUF QRF chain and capital-gains tail transfer run over the + combined frame. The remaining historical ASEC input families then run in + their declared order on source-evidenced rows: prior-year income, + relationship and Medicare inputs, housing, eligibility, pregnancy, WIC, + housing-assistance support transfer, child support, disability benefits, + workers compensation, weeks unemployed, childcare, adult care, energy + subsidy, retirement contributions/distributions, immigration, and + education inputs. +5. The pool-specific ACS transfer plan fills only still-null peer cells from + those post-assembly results. Existing measured/native cells remain + byte-for-byte unchanged, and transfer receipts record fitted and imputed + rows. +6. Schedule-D and QBI deterministic reconciliation run over that same pool. +7. The seed stage preserves existing take-up values, applies the sourced TANF and EITC mechanisms, and explicitly receipts live engine defaults used for unresolved, non-transfer-owned take-up inputs. Those defaults are not described as fitted or administrative mechanisms. -6. SSI is materialized only on an ephemeral agreement view in fixed +8. SSI is materialized only on an ephemeral agreement view in fixed household batches. Any engine defaults required solely for that calculation are separately receipted; formula output is not written into the input pool. -7. The unchanged spine-agreement gate is terminal. It uses its checked-in - registry and fixed tolerances, batches all failures, and controls the - manifest's simulation-ready status. +9. The spine-agreement gate is terminal. Its immutable pool registry is built + from the complete pool-specific transfer plan plus derived, take-up, and SSI + surfaces. Numeric columns retain the fixed incidence and conditional- + quantile tolerances. Categorical columns use a fixed weighted total- + variation-distance ceiling of `0.25`; the immigration fields are also + checked jointly so matching marginals cannot conceal incompatible pairs. + The gate batches all failures and controls the manifest's simulation-ready + status. The output H5 is a nullable, input-only, pre-calibration pool. Its companion manifest carries input pins, the assembly receipt, per-source and per-clone -counts, operator receipts, and the complete agreement result. A failed gate -writes diagnostics and a non-ready manifest and exits nonzero. Calibration -is deliberately absent; the downstream k-ladder may consume only a pool -whose terminal agreement result passed. +counts, operator receipts, and the complete agreement result. Publication +first atomically replaces any prior manifest with a non-ready tombstone, then +stages the H5 and diagnostics under one publication run ID and renames them, +and finally writes the readiness manifest. The manifest records the H5 and +diagnostics run IDs and SHA-256 digests. The readiness loader requires a green +manifest whose run ID and digests match the H5 metadata and diagnostics +payload. An interrupted, substituted, or failed publication therefore +self-reports not ready even beside stale files. A failed agreement gate writes +diagnostics and a non-ready final manifest and exits nonzero. Calibration is +deliberately absent; the downstream k-ladder +may consume only a pool whose terminal agreement result passed. ## Provenance axes @@ -150,9 +202,14 @@ Calibration is a downstream consumer boundary, not a stage in this tool. `assemble_spines(...)` is the boundary between source preparation and population operators. It receives nullable, schema-compatible peer frames and produces one combined frame before cloning, fitted transfer, derivation, -seeded assignment, simulation, or calibration. Downstream operator +seeded assignment, simulation, or calibration. Downstream pool-stage entrypoints receive that combined frame and operate on measured -characteristics without selecting behavior by source spine. +characteristics without selecting behavior by source spine. A historical +source kernel that requires CPS-only raw fields receives an ephemeral +availability projection after the combined-frame boundary is validated. Such +a projection is not published or described as a full-pool lineage state; its +declared outputs are merged back into the combined frame, whose immutable +assembly receipt remains the authority. ASEC and ACS are peer household spines. A future household source can join the same assembly contract. PUF tax detail is not a peer spine: it remains a diff --git a/packages/populace-build/src/populace/build/us_runtime/__init__.py b/packages/populace-build/src/populace/build/us_runtime/__init__.py index daecadd6..1316580c 100644 --- a/packages/populace-build/src/populace/build/us_runtime/__init__.py +++ b/packages/populace-build/src/populace/build/us_runtime/__init__.py @@ -60,7 +60,13 @@ us_alimony_summary, ) from populace.build.us_runtime.asec_checkpoint import ( + ASEC_RAW_STAGE_ARTIFACT_KIND, + ASEC_RAW_STAGE_CHECKPOINT_FILENAME, + ASEC_RAW_STAGE_OPERATOR_STATUS, + ASEC_RAW_STAGE_SCHEMA_VERSION, + ASEC_RAW_STAGE_STAGE, load_asec_pre_clone_checkpoint, + load_asec_raw_stage_checkpoint, ) from populace.build.us_runtime.asec_pool import ( AsecSource, @@ -412,6 +418,10 @@ nonzero_share, us_nonzero_shares, ) +from populace.build.us_runtime.operator_boundary import ( + PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES, + assert_operator_free_source_frame, +) from populace.build.us_runtime.org_wages import ( BLS_STATE_UNION_REPRESENTATION_RATE_2024, FLSA_EXECUTIVE_ADMINISTRATIVE_PROFESSIONAL_OCCUPATION_CODES, @@ -861,6 +871,7 @@ us_source_operation_handlers, ) from populace.build.us_runtime.spine_agreement import ( + DEFAULT_CATEGORICAL_TOTAL_VARIATION_TOLERANCE, DEFAULT_INCIDENCE_RATIO_BOUNDS, DEFAULT_QUANTILE_ENVELOPE_TOLERANCE, DEFAULT_SPINE_AGREEMENT_QUANTILES, @@ -1045,6 +1056,11 @@ from populace.frame import Frame __all__ = [ + "ASEC_RAW_STAGE_ARTIFACT_KIND", + "ASEC_RAW_STAGE_CHECKPOINT_FILENAME", + "ASEC_RAW_STAGE_OPERATOR_STATUS", + "ASEC_RAW_STAGE_SCHEMA_VERSION", + "ASEC_RAW_STAGE_STAGE", "BuildConfig", "AsecSource", "BASE_ASEC_SUPPORT_CHANNEL", @@ -1778,12 +1794,14 @@ "PUF_SOURCE_YEAR", "PUF_SOURCE_YEAR_AGI_REQUIRED_COLUMNS", "PUF_SYNTHETIC_RECID_START", + "PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES", "US_PUF_SUPPORT_FIT_NAME", "US_PUF_SUPPORT_STAGE_NAME", "US_STATE_INCOME_TAX_TARGET_SPECS", "US_STATE_INCOME_TAX_TARGET_REFERENCES", "compile_us_fiscal_target_registry", "assign_congressional_districts_to_households", + "assert_operator_free_source_frame", "build_pooled_asec_unit_frame", "clone_us_frame_for_puf_support", "congressional_district_assignment_summary", @@ -1800,6 +1818,7 @@ "load_congressional_district_vintage_crosswalk", "load_default_congressional_district_vintage_crosswalk", "load_asec_pre_clone_checkpoint", + "load_asec_raw_stage_checkpoint", "load_puf_tax_unit_donor", "normalize_district_code", "parse_baf_cd_layer", @@ -1867,6 +1886,7 @@ "us_register_consistency_gate", "us_register_contradictions", "write_us_source_coverage_diagnostics", + "DEFAULT_CATEGORICAL_TOTAL_VARIATION_TOLERANCE", "DEFAULT_INCIDENCE_RATIO_BOUNDS", "DEFAULT_QUANTILE_ENVELOPE_TOLERANCE", "DEFAULT_SPINE_AGREEMENT_QUANTILES", diff --git a/packages/populace-build/src/populace/build/us_runtime/asec_checkpoint.py b/packages/populace-build/src/populace/build/us_runtime/asec_checkpoint.py index 991cdcfe..cc1e76d6 100644 --- a/packages/populace-build/src/populace/build/us_runtime/asec_checkpoint.py +++ b/packages/populace-build/src/populace/build/us_runtime/asec_checkpoint.py @@ -1,17 +1,21 @@ -"""Bounded loading for the input-complete ASEC pre-clone checkpoint. +"""Bounded loading for ASEC producer checkpoints. -The checkpoint is produced by the outer-stage runtime after ASEC-only input -enrichment. This module validates that artifact binding and the loaded -``Frame`` boundary; it does not compute a whole-file digest. Callers that -pin input files own that separate provenance check. +The legacy checkpoint is produced by the outer-stage runtime after ASEC-only +input enrichment. The raw-stage checkpoint is a separate, auxiliary artifact +whose frame has only structural source construction and exact raw-source +mapping applied. This module validates each artifact binding and loaded +``Frame`` boundary; it does not compute a whole-file digest. Callers that pin +input files own that separate provenance check. """ from __future__ import annotations import re +from collections.abc import Mapping from pathlib import Path import numpy as np +import pandas as pd from populace.build.frame_checkpoint import load_frame_checkpoint from populace.build.outer_stage_runtime import ( @@ -19,9 +23,20 @@ FrameIdentity, frame_identity, ) +from populace.build.us_runtime.operator_boundary import ( + assert_operator_free_source_frame, +) from populace.frame import US_SCHEMA, Frame, WeightKind, Weights -__all__ = ["load_asec_pre_clone_checkpoint"] +__all__ = [ + "ASEC_RAW_STAGE_ARTIFACT_KIND", + "ASEC_RAW_STAGE_CHECKPOINT_FILENAME", + "ASEC_RAW_STAGE_OPERATOR_STATUS", + "ASEC_RAW_STAGE_SCHEMA_VERSION", + "ASEC_RAW_STAGE_STAGE", + "load_asec_pre_clone_checkpoint", + "load_asec_raw_stage_checkpoint", +] _OUTER_STAGE_ARTIFACT_KIND = "populace_outer_stage_frame" _PRE_CLONE_STAGE = "pre_clone_enrichment" @@ -38,6 +53,48 @@ } ) +ASEC_RAW_STAGE_ARTIFACT_KIND = "populace_us_asec_raw_stage" +ASEC_RAW_STAGE_CHECKPOINT_FILENAME = "asec_raw_stage.checkpoint.h5" +ASEC_RAW_STAGE_OPERATOR_STATUS = "operator_untouched" +ASEC_RAW_STAGE_SCHEMA_VERSION = 2 +ASEC_RAW_STAGE_STAGE = "raw_source_mapping" +_RAW_STAGE_BINDING_KEYS = frozenset( + { + "artifact_kind", + "identity", + "operator_status", + "pipeline_sha256", + "raw_source_mappings", + "schema_version", + "source_construction_identity", + "source_receipt", + "stage", + } +) +_RAW_SOURCE_MAPPING_COLUMNS = frozenset({"ED_VAL", "LKWEEKS"}) +_RAW_STAGE_REQUIRED_PERSON_COLUMNS = frozenset( + {"ED_VAL", "LKWEEKS", "PERIDNUM", "source_year"} +) +_RAW_SOURCE_MAPPING_KEYS = frozenset( + { + "audit", + "column", + "entity", + "join_keys", + "operation", + "source_pins", + } +) +_RAW_SOURCE_PIN_KEYS = frozenset( + { + "income_year", + "locator", + "member", + "member_sha256", + "sha256", + } +) + def load_asec_pre_clone_checkpoint( path: str | Path, @@ -81,6 +138,52 @@ def load_asec_pre_clone_checkpoint( return loaded.frame, metadata +def load_asec_raw_stage_checkpoint( + path: str | Path, +) -> tuple[Frame, dict[str, object]]: + """Load one operator-untouched ASEC raw-source-mapping checkpoint. + + Only the dedicated auxiliary producer artifact is accepted. In + particular, the legacy enriched outer-stage checkpoint cannot satisfy this + contract even if it happens to carry a structurally valid US ``Frame``. + """ + + checkpoint_path = Path(path) + loaded = load_frame_checkpoint(checkpoint_path) + metadata = _validate_raw_stage_binding( + loaded.metadata, + path=checkpoint_path, + ) + stored_identity = FrameIdentity.from_payload( + metadata["identity"], + label="ASEC raw-stage checkpoint identity", + ) + actual_identity = frame_identity(loaded.frame) + if actual_identity != stored_identity: + raise ValueError( + f"ASEC raw-stage checkpoint {checkpoint_path} Frame identity changed." + ) + _validate_asec_frame( + loaded.frame, + path=checkpoint_path, + artifact_label="ASEC raw-stage checkpoint", + ) + assert_operator_free_source_frame( + loaded.frame, + label=f"ASEC raw-stage checkpoint {checkpoint_path}", + ) + _validate_raw_stage_source_columns(loaded.frame, path=checkpoint_path) + source_construction_identity = FrameIdentity.from_payload( + metadata["source_construction_identity"], + label="ASEC raw-stage source-construction identity", + ) + metadata["identity"] = stored_identity.to_payload() + metadata["source_construction_identity"] = ( + source_construction_identity.to_payload() + ) + return loaded.frame, metadata + + def _validate_outer_stage_binding( metadata: dict[str, object], *, @@ -127,24 +230,235 @@ def _validate_outer_stage_binding( return dict(metadata) -def _validate_asec_frame(frame: Frame, *, path: Path) -> None: - if frame.schema != US_SCHEMA: +def _validate_raw_stage_binding( + metadata: dict[str, object], + *, + path: Path, +) -> dict[str, object]: + actual_keys = frozenset(metadata) + if actual_keys != _RAW_STAGE_BINDING_KEYS: + raise ValueError( + f"ASEC raw-stage checkpoint {path} has an incomplete raw-stage " + f"artifact binding (missing: " + f"{sorted(_RAW_STAGE_BINDING_KEYS - actual_keys)}; extra: " + f"{sorted(actual_keys - _RAW_STAGE_BINDING_KEYS)})." + ) + if metadata["artifact_kind"] != ASEC_RAW_STAGE_ARTIFACT_KIND: raise ValueError( - f"ASEC pre-clone checkpoint {path} must use the US entity schema." + f"ASEC raw-stage checkpoint {path} is not a dedicated raw-stage " + "ASEC artifact." ) + schema_version = metadata["schema_version"] + if ( + schema_version != ASEC_RAW_STAGE_SCHEMA_VERSION + or isinstance(schema_version, bool) + ): + raise ValueError( + f"ASEC raw-stage checkpoint {path} has an unsupported raw-stage " + "schema version." + ) + if metadata["stage"] != ASEC_RAW_STAGE_STAGE: + raise ValueError( + f"ASEC raw-stage checkpoint {path} must be bound to stage " + f"{ASEC_RAW_STAGE_STAGE!r}, got {metadata['stage']!r}." + ) + if metadata["operator_status"] != ASEC_RAW_STAGE_OPERATOR_STATUS: + raise ValueError( + f"ASEC raw-stage checkpoint {path} must declare operator_status " + f"{ASEC_RAW_STAGE_OPERATOR_STATUS!r}, got " + f"{metadata['operator_status']!r}." + ) + pipeline_sha256 = metadata["pipeline_sha256"] + if not isinstance(pipeline_sha256, str) or not _LOWERCASE_SHA256.fullmatch( + pipeline_sha256 + ): + raise ValueError( + f"ASEC raw-stage checkpoint {path} pipeline_sha256 must be a " + "lowercase SHA-256 digest." + ) + _validate_source_receipt(metadata["source_receipt"], path=path) + _validate_raw_source_mappings(metadata["raw_source_mappings"], path=path) + return dict(metadata) + + +def _validate_source_receipt(receipt: object, *, path: Path) -> None: + if not isinstance(receipt, Mapping) or receipt.get("kind") != "pooled_asec": + raise ValueError( + f"ASEC raw-stage checkpoint {path} source_receipt must describe " + "pooled_asec inputs." + ) + sources = receipt.get("sources") + if not isinstance(sources, list) or not sources: + raise ValueError( + f"ASEC raw-stage checkpoint {path} source_receipt.sources must be " + "a non-empty list." + ) + years: set[int] = set() + for index, source in enumerate(sources): + if not isinstance(source, Mapping): + raise ValueError( + f"ASEC raw-stage checkpoint {path} source_receipt.sources" + f"[{index}] must be an object." + ) + year = source.get("year") + source_path = source.get("path") + sha256 = source.get("sha256") + if ( + not isinstance(year, int) + or isinstance(year, bool) + or not isinstance(source_path, str) + or not source_path + or not isinstance(sha256, str) + or not _LOWERCASE_SHA256.fullmatch(sha256) + ): + raise ValueError( + f"ASEC raw-stage checkpoint {path} source_receipt.sources" + f"[{index}] lacks a valid year/path/SHA-256 pin." + ) + if year in years: + raise ValueError( + f"ASEC raw-stage checkpoint {path} source_receipt repeats " + f"income year {year}." + ) + years.add(year) + + +def _validate_raw_source_mappings(mappings: object, *, path: Path) -> None: + if not isinstance(mappings, Mapping): + raise ValueError( + f"ASEC raw-stage checkpoint {path} raw_source_mappings must be " + "an object." + ) + if frozenset(mappings) != _RAW_SOURCE_MAPPING_COLUMNS: + raise ValueError( + f"ASEC raw-stage checkpoint {path} raw_source_mappings must bind " + f"exactly {sorted(_RAW_SOURCE_MAPPING_COLUMNS)}." + ) + for column in sorted(_RAW_SOURCE_MAPPING_COLUMNS): + mapping = mappings[column] + if not isinstance(mapping, Mapping) or frozenset(mapping) != ( + _RAW_SOURCE_MAPPING_KEYS + ): + raise ValueError( + f"ASEC raw-stage checkpoint {path} raw_source_mappings" + f"[{column!r}] is malformed." + ) + if ( + mapping["column"] != column + or mapping["entity"] != "person" + or mapping["operation"] != "exact_source_join" + or mapping["join_keys"] != ["source_year", "PERIDNUM"] + or not isinstance(mapping["audit"], Mapping) + ): + raise ValueError( + f"ASEC raw-stage checkpoint {path} raw_source_mappings" + f"[{column!r}] does not declare the exact person-source join." + ) + pins = mapping["source_pins"] + if not isinstance(pins, list) or not pins: + raise ValueError( + f"ASEC raw-stage checkpoint {path} raw_source_mappings" + f"[{column!r}].source_pins must be a non-empty list." + ) + for index, pin in enumerate(pins): + if not isinstance(pin, Mapping) or frozenset(pin) != _RAW_SOURCE_PIN_KEYS: + raise ValueError( + f"ASEC raw-stage checkpoint {path} raw_source_mappings" + f"[{column!r}].source_pins[{index}] is malformed." + ) + sha256 = pin["sha256"] + member_sha256 = pin["member_sha256"] + if ( + not isinstance(pin["income_year"], int) + or isinstance(pin["income_year"], bool) + or not isinstance(pin["locator"], str) + or not pin["locator"] + or not isinstance(pin["member"], str) + or not pin["member"] + or not isinstance(sha256, str) + or not _LOWERCASE_SHA256.fullmatch(sha256) + or not isinstance(member_sha256, str) + or not _LOWERCASE_SHA256.fullmatch(member_sha256) + ): + raise ValueError( + f"ASEC raw-stage checkpoint {path} raw_source_mappings" + f"[{column!r}].source_pins[{index}] lacks immutable pins." + ) + + +def _validate_raw_stage_source_columns(frame: Frame, *, path: Path) -> None: + person = frame.table("person") + missing = sorted(_RAW_STAGE_REQUIRED_PERSON_COLUMNS - set(person)) + if missing: + raise ValueError( + f"ASEC raw-stage checkpoint {path} is not input-complete; missing " + f"raw person column(s): {missing}." + ) + + source_year = pd.to_numeric(person["source_year"], errors="coerce").to_numpy( + dtype=np.float64 + ) + if ( + not np.isfinite(source_year).all() + or not np.equal(source_year, np.floor(source_year)).all() + ): + raise ValueError( + f"ASEC raw-stage checkpoint {path} source_year must be complete " + "finite integers." + ) + + peridnum = person["PERIDNUM"] + valid_peridnum = peridnum.notna() + if pd.api.types.is_string_dtype(peridnum.dtype) or peridnum.dtype == object: + valid_peridnum &= ( + peridnum.astype("string").str.strip().ne("").fillna(False) + ) + if not valid_peridnum.all(): + raise ValueError( + f"ASEC raw-stage checkpoint {path} PERIDNUM must be complete and " + "nonempty." + ) + + education = pd.to_numeric(person["ED_VAL"], errors="coerce").to_numpy( + dtype=np.float64 + ) + if not (np.isfinite(education) & (education >= 0.0)).all(): + raise ValueError( + f"ASEC raw-stage checkpoint {path} ED_VAL must be complete, finite, " + "and nonnegative." + ) + + weeks = pd.to_numeric(person["LKWEEKS"], errors="coerce").to_numpy( + dtype=np.float64 + ) + valid_weeks = np.isfinite(weeks) & np.equal(weeks, np.floor(weeks)) + valid_weeks &= (weeks == -1.0) | ((weeks >= 0.0) & (weeks <= 52.0)) + if not valid_weeks.all(): + raise ValueError( + f"ASEC raw-stage checkpoint {path} LKWEEKS must be complete integers " + "in {-1, 0, ..., 52}." + ) + + +def _validate_asec_frame( + frame: Frame, + *, + path: Path, + artifact_label: str = "ASEC pre-clone checkpoint", +) -> None: + if frame.schema != US_SCHEMA: + raise ValueError(f"{artifact_label} {path} must use the US entity schema.") if frame.weighted_entities != ("household",): raise ValueError( - f"ASEC pre-clone checkpoint {path} must carry household weights only; " + f"{artifact_label} {path} must carry household weights only; " f"got weighted entities {list(frame.weighted_entities)}." ) weights = frame.weights_for("household") if not isinstance(weights, Weights) or not isinstance(weights.kind, WeightKind): - raise ValueError( - f"ASEC pre-clone checkpoint {path} household weights must be typed." - ) + raise ValueError(f"{artifact_label} {path} household weights must be typed.") values = weights.values if not np.isfinite(values).all() or not (values > 0.0).all(): raise ValueError( - f"ASEC pre-clone checkpoint {path} household weights must be " + f"{artifact_label} {path} household weights must be " "strictly positive and finite." ) diff --git a/packages/populace-build/src/populace/build/us_runtime/h5_io.py b/packages/populace-build/src/populace/build/us_runtime/h5_io.py index 320afe01..4e8f1e4d 100644 --- a/packages/populace-build/src/populace/build/us_runtime/h5_io.py +++ b/packages/populace-build/src/populace/build/us_runtime/h5_io.py @@ -9,10 +9,13 @@ from __future__ import annotations +import hashlib import json import os +import re import uuid import warnings +from collections.abc import Mapping from pathlib import Path import numpy as np @@ -23,13 +26,28 @@ __all__ = [ "LEGACY_NULLABLE_STAGING_ARTIFACT_KIND", + "US_MULTISPINE_AGREEMENT_DIAGNOSTICS_ARTIFACT_KIND", + "US_MULTISPINE_POOL_H5_ARTIFACT_KIND", + "US_MULTISPINE_POOL_MANIFEST_ARTIFACT_KIND", + "US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION", "load_legacy_calibrated_us_h5", + "load_simulation_ready_us_multispine_pool_manifest", + "read_nullable_us_h5_metadata", "write_nullable_us_h5", ] LEGACY_NULLABLE_STAGING_ARTIFACT_KIND = "nullable_precalibration_staging_h5" +US_MULTISPINE_POOL_MANIFEST_ARTIFACT_KIND = ( + "populace_us_multispine_pool_manifest" +) +US_MULTISPINE_POOL_H5_ARTIFACT_KIND = "populace_us_multispine_input_pool" +US_MULTISPINE_AGREEMENT_DIAGNOSTICS_ARTIFACT_KIND = ( + "populace_us_multispine_agreement_diagnostics" +) +US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION = 3 _METADATA_KEY = "_populace_staging_metadata" _TIME_PERIOD_KEY = "_time_period" +_LOWERCASE_SHA256 = re.compile(r"[0-9a-f]{64}") def load_legacy_calibrated_us_h5(path: str | Path) -> Frame: @@ -68,6 +86,152 @@ def load_legacy_calibrated_us_h5(path: str | Path) -> Frame: ) +def load_simulation_ready_us_multispine_pool_manifest( + path: str | Path, +) -> dict[str, object]: + """Validate and return one ready manifest bound to its H5 and diagnostics. + + The manifest is the readiness authority. A caller cannot treat an H5 as + ready merely because it exists: the manifest, nested artifact receipts, + H5 metadata, diagnostics, and file digests must all bind the same + publication run. + """ + + manifest_path = Path(path) + manifest = _read_json_object(manifest_path, label="pool manifest") + if ( + manifest.get("artifact_kind") + != US_MULTISPINE_POOL_MANIFEST_ARTIFACT_KIND + or manifest.get("schema_version") + != US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION + ): + raise ValueError( + f"US multispine pool manifest {manifest_path} has an unsupported " + "artifact binding." + ) + if ( + manifest.get("simulation_ready") is not True + or manifest.get("status") != "simulation_ready" + ): + raise ValueError( + f"US multispine pool manifest {manifest_path} is not " + "simulation-ready." + ) + publication_run_id = _publication_run_id( + manifest.get("publication_run_id"), + label=f"US multispine pool manifest {manifest_path}", + ) + + pool_receipt = _mapping( + manifest.get("pool_h5"), + label=f"US multispine pool manifest {manifest_path}.pool_h5", + ) + if pool_receipt.get("artifact_kind") != US_MULTISPINE_POOL_H5_ARTIFACT_KIND: + raise ValueError( + f"US multispine pool manifest {manifest_path} names the wrong H5 " + "artifact kind." + ) + _require_matching_run_id( + pool_receipt, + publication_run_id, + label=f"US multispine pool manifest {manifest_path}.pool_h5", + ) + pool_path = _artifact_path( + pool_receipt, + label=f"US multispine pool manifest {manifest_path}.pool_h5", + ) + _require_matching_sha256( + pool_path, + pool_receipt, + label=f"US multispine pool manifest {manifest_path}.pool_h5", + ) + h5_metadata = read_nullable_us_h5_metadata(pool_path) + if h5_metadata.get("artifact_kind") != US_MULTISPINE_POOL_H5_ARTIFACT_KIND: + raise ValueError( + f"US multispine pool H5 {pool_path} has the wrong artifact kind." + ) + if h5_metadata.get("publication_run_id") != publication_run_id: + raise ValueError( + f"US multispine pool H5 {pool_path} publication run ID does not " + "match its manifest." + ) + + diagnostics_receipt = _mapping( + manifest.get("agreement_diagnostics"), + label=( + f"US multispine pool manifest {manifest_path}.agreement_diagnostics" + ), + ) + _require_matching_run_id( + diagnostics_receipt, + publication_run_id, + label=( + f"US multispine pool manifest {manifest_path}.agreement_diagnostics" + ), + ) + diagnostics_path = _artifact_path( + diagnostics_receipt, + label=( + f"US multispine pool manifest {manifest_path}.agreement_diagnostics" + ), + ) + _require_matching_sha256( + diagnostics_path, + diagnostics_receipt, + label=( + f"US multispine pool manifest {manifest_path}.agreement_diagnostics" + ), + ) + diagnostics = _read_json_object( + diagnostics_path, + label="pool agreement diagnostics", + ) + if ( + diagnostics.get("artifact_kind") + != US_MULTISPINE_AGREEMENT_DIAGNOSTICS_ARTIFACT_KIND + or diagnostics.get("schema_version") + != US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION + or diagnostics.get("simulation_ready") is not True + or diagnostics.get("publication_run_id") != publication_run_id + ): + raise ValueError( + f"US multispine pool diagnostics {diagnostics_path} do not match " + "the ready manifest publication." + ) + return manifest + + +def read_nullable_us_h5_metadata(path: str | Path) -> dict[str, object]: + """Read and validate the single JSON artifact-metadata row from an H5.""" + + h5_path = Path(path) + if not h5_path.is_file(): + raise FileNotFoundError(f"Nullable US H5 is not a file: {h5_path}") + with pd.HDFStore(h5_path, mode="r") as store: + try: + raw_metadata = store[_METADATA_KEY] + except KeyError as exc: + raise ValueError( + f"Nullable US H5 {h5_path} has no artifact metadata." + ) from exc + if len(raw_metadata) != 1: + raise ValueError( + f"Nullable US H5 {h5_path} must carry exactly one artifact " + "metadata row." + ) + try: + metadata = json.loads(str(raw_metadata.iloc[0])) + except (TypeError, ValueError) as exc: + raise ValueError( + f"Nullable US H5 {h5_path} artifact metadata is not valid JSON." + ) from exc + if not isinstance(metadata, dict): + raise ValueError( + f"Nullable US H5 {h5_path} artifact metadata must be a JSON object." + ) + return metadata + + def write_nullable_us_h5( frame: Frame, path: str | Path, @@ -245,3 +409,62 @@ def _artifact_metadata( if publication_run_id is not None: metadata["publication_run_id"] = publication_run_id return metadata + + +def _read_json_object(path: Path, *, label: str) -> dict[str, object]: + try: + payload = json.loads(Path(path).read_text(encoding="utf-8")) + except (OSError, TypeError, ValueError) as exc: + raise ValueError(f"{label} {path} is not readable valid JSON.") from exc + if not isinstance(payload, dict): + raise ValueError(f"{label} {path} must contain a JSON object.") + return payload + + +def _mapping(value: object, *, label: str) -> Mapping[str, object]: + if not isinstance(value, Mapping): + raise ValueError(f"{label} must be an object.") + return value + + +def _publication_run_id(value: object, *, label: str) -> str: + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"{label} must name a non-empty publication run ID.") + return value + + +def _require_matching_run_id( + receipt: Mapping[str, object], + publication_run_id: str, + *, + label: str, +) -> None: + if receipt.get("publication_run_id") != publication_run_id: + raise ValueError(f"{label} publication run ID does not match the manifest.") + + +def _artifact_path(receipt: Mapping[str, object], *, label: str) -> Path: + raw_path = receipt.get("path") + if not isinstance(raw_path, str) or not raw_path.strip(): + raise ValueError(f"{label}.path must be a non-empty string.") + path = Path(raw_path) + if not path.is_file(): + raise ValueError(f"{label}.path is not a file: {path}") + return path + + +def _require_matching_sha256( + path: Path, + receipt: Mapping[str, object], + *, + label: str, +) -> None: + expected = receipt.get("sha256") + if not isinstance(expected, str) or not _LOWERCASE_SHA256.fullmatch(expected): + raise ValueError(f"{label}.sha256 must be a lowercase SHA-256 digest.") + digest = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + if digest.hexdigest() != expected: + raise ValueError(f"{label} SHA-256 does not match the published artifact.") diff --git a/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py b/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py index 1971e572..5ecc54a8 100644 --- a/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py +++ b/packages/populace-build/src/populace/build/us_runtime/multispine_pool.py @@ -21,20 +21,64 @@ from populace.build.gates import GateResult from populace.build.us_runtime.acs_transfer import ( + ACS_NATIVE_PERSON_INPUTS, TargetFamilies, declared_acs_transfer_target_families, derive_acs_schedule_d_capital_gain_distributions, ) +from populace.build.us_runtime.adult_care import with_us_adult_care_inputs +from populace.build.us_runtime.child_support import with_us_child_support_inputs +from populace.build.us_runtime.childcare import with_us_childcare_inputs +from populace.build.us_runtime.cps_carried import derive_us_cps_carried_inputs +from populace.build.us_runtime.disability_benefits import ( + with_us_disability_benefits, +) +from populace.build.us_runtime.education_inputs import with_us_education_inputs +from populace.build.us_runtime.eligibility_inputs import ( + with_us_eligibility_inputs, +) +from populace.build.us_runtime.energy_subsidy import ( + with_us_energy_subsidy_input, +) +from populace.build.us_runtime.housing_inputs import ( + impute_us_housing_assistance_to_puf_support, + with_us_housing_inputs, +) +from populace.build.us_runtime.immigration import with_us_immigration_inputs +from populace.build.us_runtime.medicare_take_up import ( + with_us_medicare_take_up_input, +) +from populace.build.us_runtime.operator_boundary import ( + PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES, +) +from populace.build.us_runtime.pregnancy import with_us_pregnancy_inputs +from populace.build.us_runtime.prior_year_income import ( + with_us_prior_year_income_inputs, +) from populace.build.us_runtime.puf_support import clone_us_frame_for_puf_support from populace.build.us_runtime.qbi_inputs import ( US_QBI_OUTPUT_COLUMNS, with_us_qbi_input_reconciliation, ) -from populace.build.us_runtime.spine_agreement import spine_agreement_gate +from populace.build.us_runtime.relationship_inputs import ( + with_us_relationship_inputs, +) +from populace.build.us_runtime.retirement_contributions import ( + with_us_retirement_contribution_inputs, +) +from populace.build.us_runtime.retirement_distributions import ( + with_us_retirement_distribution_inputs, +) +from populace.build.us_runtime.spine_agreement import ( + default_spine_agreement_registry, + spine_agreement_gate, +) from populace.build.us_runtime.spine_assembly import assemble_spines from populace.build.us_runtime.support_provenance import ( + SPINE_ASSEMBLY_MANIFEST_KEY, spine_assembly_receipt, spine_provenance_counts, + support_clone_index_column, validate_assembly_provenance, ) from populace.build.us_runtime.take_up import with_us_take_up_inputs @@ -42,6 +86,11 @@ TakeUpProgram, load_take_up_contract, ) +from populace.build.us_runtime.weeks_unemployed import with_us_weeks_unemployed +from populace.build.us_runtime.wic_claim import with_us_wic_claim_input +from populace.build.us_runtime.workers_compensation import ( + with_us_workers_compensation, +) from populace.frame import Frame __all__ = [ @@ -49,12 +98,16 @@ "POOL_OPERATOR_ORDER", "POOL_RANDOM_SEED", "POOL_SIMULATION_HOUSEHOLD_BATCH_SIZE", + "POOL_SOURCE_OPERATOR_ORDER", + "POOL_SPINE_AGREEMENT_REGISTRY", "POOL_TIME_PERIOD", "MultispinePoolResult", "PoolStageOutput", + "complete_multispine_source_inputs", "derive_multispine_pool_inputs", "materialize_multispine_agreement_outputs", "pool_transfer_target_families", + "prepare_multispine_puf_predictors", "run_multispine_pool_path", "seed_multispine_pool_inputs", ] @@ -76,6 +129,34 @@ ) """The executable pool-build order, including the terminal QA evaluation.""" +POOL_SOURCE_OPERATOR_ORDER = ( + "derive_us_cps_carried_inputs", + "with_us_prior_year_income_inputs", + "with_us_relationship_inputs", + "with_us_medicare_take_up_input", + "with_us_housing_inputs", + "with_us_eligibility_inputs", + "with_us_pregnancy_inputs", + "with_us_wic_claim_input", + "impute_us_housing_assistance_to_puf_support", + "with_us_child_support_inputs", + "with_us_disability_benefits", + "with_us_workers_compensation", + "with_us_weeks_unemployed", + "with_us_childcare_inputs", + "with_us_adult_care_inputs", + "with_us_energy_subsidy_input", + "with_us_retirement_contribution_inputs", + "with_us_retirement_distribution_inputs", + "with_us_immigration_inputs", + "with_us_education_inputs", +) +"""Migrated source-input order. + +CPS predictor preparation runs before primary QRF; every remaining entry runs +post-tail on the already assembled and cloned population. +""" + POOL_RANDOM_SEED = 0 """Fixed seed shared by pool imputations and seeded input stages.""" @@ -137,22 +218,583 @@ def simulation_ready(self) -> bool: type PoolOperator = Callable[[Frame], PoolStageOutput] type AgreementGate = Callable[[Frame], GateResult] +type SourceFrameOperator = Callable[[Frame], Frame] + +_CPS_SOURCE_EVIDENCE_COLUMN = "PERIDNUM" +_SOURCE_OPERATOR_FAMILIES: Mapping[str, str] = { + "derive_us_cps_carried_inputs": "cps_carried", + "with_us_prior_year_income_inputs": "prior_year_income", + "with_us_relationship_inputs": "relationship_inputs", + "with_us_medicare_take_up_input": "medicare_take_up", + "with_us_housing_inputs": "housing_inputs", + "with_us_eligibility_inputs": "eligibility_inputs", + "with_us_pregnancy_inputs": "pregnancy", + "with_us_wic_claim_input": "wic_claim", + "impute_us_housing_assistance_to_puf_support": "housing_assistance", + "with_us_child_support_inputs": "child_support", + "with_us_disability_benefits": "disability_benefits", + "with_us_workers_compensation": "workers_compensation", + "with_us_weeks_unemployed": "weeks_unemployed", + "with_us_childcare_inputs": "childcare", + "with_us_adult_care_inputs": "adult_care", + "with_us_energy_subsidy_input": "energy_subsidy", + "with_us_retirement_contribution_inputs": "retirement_contributions", + "with_us_retirement_distribution_inputs": "retirement_distributions", + "with_us_immigration_inputs": "immigration", + "with_us_education_inputs": "education_inputs", +} +_FORMULA_OWNED_SOURCE_OUTPUTS: Mapping[str, frozenset[str]] = { + "person": frozenset({"employment_income_last_year"}), +} +_POOL_NATIVE_COMPLETE_OUTPUTS: Mapping[str, frozenset[str]] = { + "person": frozenset( + { + *ACS_NATIVE_PERSON_INPUTS, + "age", + "is_female", + "is_household_head", + } + ), + "household": frozenset({"tenure_type"}), + "spm_unit": frozenset({"spm_unit_tenure_type"}), +} def pool_transfer_target_families() -> TargetFamilies: - """Return the fixed raw-preserving QRF transfer plan. - - The #581 default agreement registry supplements this declaration with the - complete take-up inventory and formula-owned SSI. Take-up inputs not owned - by the declared QRF are handled together in the later seed stage, where - sourced TANF/EITC draws and explicitly disclosed engine defaults remain - distinguishable in the receipt. + """Return the fixed pool-only raw-preserving QRF transfer plan. + + The legacy declaration remains unchanged. This pool-local copy adds every + persisted historical source-operator output that ACS does not map natively + and the legacy plan does not already own. A target appears in exactly one + family, so transfer provenance stays unambiguous. Formula-owned outputs are + never transfer targets. + + The #581 agreement registry supplements this plan with the complete take-up + inventory and formula-owned SSI. Take-up inputs not owned by QRF remain in + the later seed stage, where sourced draws and disclosed engine defaults + remain distinguishable in the receipt. """ - return { + plan: dict[str, dict[str, tuple[str, ...]]] = { entity: {family: tuple(columns) for family, columns in families.items()} for entity, families in declared_acs_transfer_target_families().items() } + declared = { + column + for families in plan.values() + for columns in families.values() + for column in columns + } + for operator_name in POOL_SOURCE_OPERATOR_ORDER: + family = _SOURCE_OPERATOR_FAMILIES[operator_name] + additions: dict[str, tuple[str, ...]] = {} + for entity, columns in PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES[family].items(): + excluded = ( + declared + | set(_POOL_NATIVE_COMPLETE_OUTPUTS.get(entity, ())) + | set(_FORMULA_OWNED_SOURCE_OUTPUTS.get(entity, ())) + ) + targets = tuple(sorted(set(columns) - excluded)) + if targets: + additions[entity] = targets + declared.update(targets) + for entity, targets in additions.items(): + plan.setdefault(entity, {})[f"source_operator_{family}"] = targets + return plan + + +POOL_SPINE_AGREEMENT_REGISTRY = default_spine_agreement_registry( + pool_transfer_target_families() +) +"""Immutable pool charter: transfers, derived leaves, take-up, and SSI.""" + + +def prepare_multispine_puf_predictors(frame: Frame) -> PoolStageOutput: + """Derive CPS-carried primary-QRF predictors after assembly and cloning. + + ``PERIDNUM`` is raw CPS evidence and is absent from the harmonized ACS + source. Only rows carrying that raw evidence enter the historical + derivation. Explicit output families are merged back by structural entity + ID, leaving unavailable peer-spine cells nullable and preserving native + non-null cells. + """ + + return _run_source_operator_chain( + frame, + operator_names=(POOL_SOURCE_OPERATOR_ORDER[0],), + operators={ + POOL_SOURCE_OPERATOR_ORDER[0]: derive_us_cps_carried_inputs, + }, + ) + + +def complete_multispine_source_inputs( + frame: Frame, + *, + acs_rent_donor: pd.DataFrame, +) -> PoolStageOutput: + """Run the remaining historical source chain on the assembled clone pool. + + The function is intentionally fixed-seed/fixed-period. Each operator runs + over the CPS-evidenced portion of the already assembled and cloned frame; + its declared output family alone is merged into the whole pool. This + retains ACS native measurements, leaves unavailable cells null for the + subsequent declared transfer, and keeps assembly metadata untouched. + """ + + operators: Mapping[str, SourceFrameOperator] = { + "with_us_prior_year_income_inputs": lambda current: ( + with_us_prior_year_income_inputs( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ) + ), + "with_us_relationship_inputs": lambda current: with_us_relationship_inputs( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ), + "with_us_medicare_take_up_input": lambda current: ( + with_us_medicare_take_up_input( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ) + ), + "with_us_housing_inputs": lambda current: with_us_housing_inputs( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + acs_rent_donor=acs_rent_donor, + ), + "with_us_eligibility_inputs": lambda current: with_us_eligibility_inputs( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ), + "with_us_pregnancy_inputs": lambda current: with_us_pregnancy_inputs( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ), + "with_us_wic_claim_input": lambda current: with_us_wic_claim_input( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ), + "impute_us_housing_assistance_to_puf_support": lambda current: ( + impute_us_housing_assistance_to_puf_support( + current, + seed=POOL_RANDOM_SEED, + ) + ), + "with_us_child_support_inputs": lambda current: with_us_child_support_inputs( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ), + "with_us_disability_benefits": lambda current: with_us_disability_benefits( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ), + "with_us_workers_compensation": lambda current: ( + with_us_workers_compensation( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ) + ), + "with_us_weeks_unemployed": lambda current: with_us_weeks_unemployed( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ), + "with_us_childcare_inputs": lambda current: with_us_childcare_inputs( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ), + "with_us_adult_care_inputs": lambda current: with_us_adult_care_inputs( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ), + "with_us_energy_subsidy_input": lambda current: with_us_energy_subsidy_input( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ), + "with_us_retirement_contribution_inputs": lambda current: ( + with_us_retirement_contribution_inputs( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ) + ), + "with_us_retirement_distribution_inputs": lambda current: ( + with_us_retirement_distribution_inputs( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + force_puf_imputation=True, + ) + ), + "with_us_immigration_inputs": lambda current: with_us_immigration_inputs( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ), + "with_us_education_inputs": lambda current: with_us_education_inputs( + current, + seed=POOL_RANDOM_SEED, + time_period=POOL_TIME_PERIOD, + ), + } + return _run_source_operator_chain( + frame, + operator_names=POOL_SOURCE_OPERATOR_ORDER[1:], + operators=operators, + ) + + +def _run_source_operator_chain( + frame: Frame, + *, + operator_names: tuple[str, ...], + operators: Mapping[str, SourceFrameOperator], + output_families: Mapping[ + str, + Mapping[str, frozenset[str]], + ] = PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES, +) -> PoolStageOutput: + """Run an injectable source-available chain and merge declared outputs.""" + + if not isinstance(frame, Frame): + raise TypeError( + "Multispine source operators require a Frame, got " + f"{type(frame).__name__}." + ) + expected_operators = set(operator_names) + if set(operators) != expected_operators: + raise ValueError( + "Multispine source operator mapping must exactly match the requested " + f"order; missing={sorted(expected_operators - set(operators))}, " + f"unexpected={sorted(set(operators) - expected_operators)}." + ) + invalid = [name for name, operator in operators.items() if not callable(operator)] + if invalid: + raise TypeError(f"Multispine source operator(s) are not callable: {invalid}.") + + _assert_source_operator_boundary(frame) + current = frame + receipts: list[dict[str, object]] = [] + for order_index, operator_name in enumerate(operator_names): + family = _SOURCE_OPERATOR_FAMILIES.get(operator_name, operator_name) + if family not in output_families: + raise ValueError( + f"Multispine source operator {operator_name!r} has no declared " + f"output family {family!r}." + ) + declared_outputs = _persisted_source_outputs( + output_families[family], + ) + available_mask = _cps_source_evidence_mask(current) + available = _source_available_projection(current, available_mask) + available = _without_unavailable_output_columns( + available, + declared_outputs, + ) + before_rows = _frame_row_counts(current) + available_rows = _frame_row_counts(available) + outcome = operators[operator_name](available) + if not isinstance(outcome, Frame): + raise TypeError( + f"Multispine source operator {operator_name!r} must return Frame, " + f"got {type(outcome).__name__}." + ) + output_rows = _frame_row_counts(outcome) + if output_rows != available_rows: + raise ValueError( + f"Multispine source operator {operator_name!r} changed entity row " + f"counts: input={available_rows}, output={output_rows}." + ) + _assert_source_operator_structure( + available, + outcome, + operator_name=operator_name, + ) + current, merged_rows = _merge_source_operator_outputs( + current, + outcome, + declared_outputs, + operator_name=operator_name, + ) + after_rows = _frame_row_counts(current) + if after_rows != before_rows: + raise AssertionError( + f"Multispine source output merge changed pool row counts at " + f"{operator_name!r}: input={before_rows}, output={after_rows}." + ) + receipts.append( + { + "order_index": order_index, + "operator": operator_name, + "family": family, + "pool_input_rows": before_rows, + "cps_available_rows": available_rows, + "operator_output_rows": output_rows, + "merged_rows": merged_rows, + "operator_projection": { + "selection": _CPS_SOURCE_EVIDENCE_COLUMN, + "lineage_state_persisted": False, + }, + "output_columns": { + entity: sorted(columns) + for entity, columns in declared_outputs.items() + if columns + }, + } + ) + return PoolStageOutput( + current, + { + "operator_order": list(operator_names), + "cps_source_evidence": { + "column": _CPS_SOURCE_EVIDENCE_COLUMN, + "person_rows": int(_cps_source_evidence_mask(frame).sum()), + }, + "suboperators": receipts, + }, + ) + + +def _assert_source_operator_boundary(frame: Frame) -> None: + manifest = frame.metadata.get(SPINE_ASSEMBLY_MANIFEST_KEY) + if not isinstance(manifest, Mapping): + raise ValueError( + "Multispine source operators require the immutable spine assembly " + "manifest before any source derivation." + ) + person = frame.table(frame.schema.person_entity) + clone_column = support_clone_index_column(frame.schema.person_entity) + if clone_column not in person: + raise ValueError( + "Multispine source operators require post-assembly clone provenance; " + f"missing {clone_column!r}." + ) + clone_index = pd.to_numeric(person[clone_column], errors="coerce") + if clone_index.isna().any(): + raise ValueError( + f"Multispine source clone provenance {clone_column!r} must be integral." + ) + clone_values = clone_index.to_numpy(dtype=np.float64) + if ( + (clone_values < 0.0).any() + or not np.equal(clone_values, np.floor(clone_values)).all() + or not np.any(clone_values == 0.0) + or not np.any(clone_values > 0.0) + ): + raise ValueError( + "Multispine source operators require both native and cloned rows with " + "nonnegative integral clone provenance." + ) + _cps_source_evidence_mask(frame) + + +def _cps_source_evidence_mask(frame: Frame) -> pd.Series: + """Select CPS lineage only from a raw column unavailable on ACS.""" + + person = frame.table(frame.schema.person_entity) + if _CPS_SOURCE_EVIDENCE_COLUMN not in person: + raise ValueError( + "Multispine source operators require raw CPS evidence column " + f"{_CPS_SOURCE_EVIDENCE_COLUMN!r}." + ) + evidence = person[_CPS_SOURCE_EVIDENCE_COLUMN] + available = evidence.notna() + if pd.api.types.is_string_dtype(evidence.dtype) or evidence.dtype == object: + available &= evidence.astype("string").str.strip().ne("").fillna(False) + if not available.any(): + raise ValueError( + "Multispine source operators found no CPS-evidenced person rows in " + f"{_CPS_SOURCE_EVIDENCE_COLUMN!r}." + ) + clone_column = support_clone_index_column(frame.schema.person_entity) + clone_index = pd.to_numeric(person[clone_column], errors="coerce") + evidenced_clones = set(clone_index.loc[available].astype(int).tolist()) + if 0 not in evidenced_clones or not any(index > 0 for index in evidenced_clones): + raise ValueError( + "Raw CPS evidence must cover both native and cloned rows before " + "source operators run." + ) + return available.astype(bool) + + +def _source_available_projection(frame: Frame, person_mask: pd.Series) -> Frame: + """Build an ephemeral CPS-only kernel input without a false pool receipt. + + The public source-chain boundary receives and validates the fully assembled + clone pool. Historical source operators cannot safely consume rows lacking + their raw CPS inputs, so their internal kernel runs on this structural + projection. Assembly metadata and mass history describe the full pool and + therefore must not be attached to the subset. Only declared outputs are + merged back into the still-receipted full pool. + """ + + selected = frame.select(person_mask) + return Frame( + { + entity: selected.table(entity).copy() + for entity in selected.entities + }, + selected.schema, + { + entity: selected.weights_for(entity) + for entity in selected.weighted_entities + }, + selected.strata, + ) + + +def _persisted_source_outputs( + outputs: Mapping[str, frozenset[str]], +) -> dict[str, frozenset[str]]: + return { + entity: frozenset( + set(columns) - set(_FORMULA_OWNED_SOURCE_OUTPUTS.get(entity, ())) + ) + for entity, columns in outputs.items() + } + + +def _assert_source_operator_structure( + before: Frame, + after: Frame, + *, + operator_name: str, +) -> None: + if after.metadata != before.metadata or after.mass_log != before.mass_log: + raise ValueError( + f"Multispine source operator {operator_name!r} changed immutable " + "assembly metadata or mass history." + ) + for entity in before.entities: + entity_id = before.schema.entity_id_column(entity) + before_ids = before.table(entity)[entity_id] + after_ids = after.table(entity)[entity_id] + if ( + after_ids.duplicated().any() + or set(after_ids.tolist()) != set(before_ids.tolist()) + ): + raise ValueError( + f"Multispine source operator {operator_name!r} changed structural " + f"{entity_id!r} values." + ) + + +def _without_unavailable_output_columns( + frame: Frame, + outputs: Mapping[str, frozenset[str]], +) -> Frame: + """Remove union-created all-null outputs before an available-source run.""" + + tables = {entity: frame.table(entity).copy() for entity in frame.entities} + dropped = False + for entity, columns in outputs.items(): + if entity not in tables: + continue + unavailable = [ + column + for column in columns + if column in tables[entity] and tables[entity][column].isna().all() + ] + if unavailable: + tables[entity] = tables[entity].drop(columns=unavailable) + dropped = True + if not dropped: + return frame + return Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + + +def _merge_source_operator_outputs( + pool: Frame, + operated: Frame, + outputs: Mapping[str, frozenset[str]], + *, + operator_name: str, +) -> tuple[Frame, dict[str, int]]: + """Merge only one operator's explicit family by entity ID.""" + + tables = {entity: pool.table(entity).copy() for entity in pool.entities} + merged_rows: dict[str, int] = {} + for entity, columns in outputs.items(): + if not columns: + continue + if entity not in pool.entities or entity not in operated.entities: + raise ValueError( + f"Multispine source operator {operator_name!r} declares outputs " + f"for absent entity {entity!r}." + ) + target = tables[entity] + source = operated.table(entity) + entity_id = pool.schema.entity_id_column(entity) + if entity_id not in target or entity_id not in source: + raise ValueError( + f"Multispine source operator {operator_name!r} cannot align " + f"{entity!r} without {entity_id!r}." + ) + if source[entity_id].duplicated().any(): + raise ValueError( + f"Multispine source operator {operator_name!r} returned duplicate " + f"{entity_id!r} values." + ) + missing_outputs = sorted(set(columns) - set(source.columns)) + if missing_outputs: + raise ValueError( + f"Multispine source operator {operator_name!r} did not emit its " + f"declared {entity!r} output(s): {missing_outputs}." + ) + source_by_id = source.set_index(entity_id) + target_ids = target[entity_id] + eligible = target_ids.isin(source_by_id.index) + if int(eligible.sum()) != len(source): + raise ValueError( + f"Multispine source operator {operator_name!r} output IDs do not " + f"align one-to-one with the {entity!r} pool." + ) + for column in sorted(columns): + aligned = source_by_id[column].reindex(target_ids) + if column not in target: + target[column] = aligned.to_numpy() + else: + positions = np.flatnonzero(eligible.to_numpy()) + target.loc[target.index[positions], column] = aligned.iloc[ + positions + ].to_numpy() + merged_rows[entity] = int(eligible.sum()) + + merged = Frame( + tables, + pool.schema, + {entity: pool.weights_for(entity) for entity in pool.weighted_entities}, + pool.strata, + mass_log=pool.mass_log, + metadata=pool.metadata, + ) + return merged, merged_rows + + +def _frame_row_counts(frame: Frame) -> dict[str, int]: + return {entity: int(len(frame.table(entity))) for entity in frame.entities} def derive_multispine_pool_inputs(frame: Frame) -> PoolStageOutput: @@ -600,7 +1242,7 @@ def run_multispine_pool_path( ``agreement_gate`` is an injection seam for small synthetic tests only. Production callers omit it, which invokes :func:`~populace.build.us_runtime.spine_agreement.spine_agreement_gate` - with its fixed registry and tolerances. + with the immutable pool-specific registry and fixed tolerances. """ operators = { @@ -660,8 +1302,14 @@ def run_multispine_pool_path( ) receipts["simulate"] = dict(simulated.receipt) - gate_operator = spine_agreement_gate if agreement_gate is None else agreement_gate - agreement = gate_operator(simulated.frame) + agreement = ( + spine_agreement_gate( + simulated.frame, + registry=POOL_SPINE_AGREEMENT_REGISTRY, + ) + if agreement_gate is None + else agreement_gate(simulated.frame) + ) if not isinstance(agreement, GateResult): raise TypeError( "Pool agreement operator must return GateResult, got " diff --git a/packages/populace-build/src/populace/build/us_runtime/operator_boundary.py b/packages/populace-build/src/populace/build/us_runtime/operator_boundary.py new file mode 100644 index 00000000..6c81ce97 --- /dev/null +++ b/packages/populace-build/src/populace/build/us_runtime/operator_boundary.py @@ -0,0 +1,471 @@ +"""Fail-closed boundary between raw source mapping and US operators. + +The registry is deliberately explicit and entity-scoped. A raw ASEC frame +must contain none of these canonical outputs. An ACS frame may carry a +canonical-looking column only when the exact native-mapping receipt emitted by +``map_acs_native_inputs`` accounts for the column, its entity, row counts, and +raw source columns. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import Any + +import pandas as pd + +from populace.build.us_runtime.acs_transfer import ACS_DERIVED_TRANSFER_INPUTS +from populace.build.us_runtime.adult_care import US_ADULT_CARE_OUTPUT_COLUMNS +from populace.build.us_runtime.child_support import ( + US_CHILD_SUPPORT_OUTPUT_COLUMNS, +) +from populace.build.us_runtime.childcare import US_CHILDCARE_OUTPUT_COLUMNS +from populace.build.us_runtime.congressional_district_geography import ( + CONGRESSIONAL_DISTRICT_GEOID_COLUMN, +) +from populace.build.us_runtime.cps_carried import ( + CPS_CARRIED_FORMULA_OWNED_COLUMNS, + CPS_CARRIED_PERSON_INPUTS, + CPS_CARRIED_SPM_UNIT_INPUTS, +) +from populace.build.us_runtime.disability_benefits import ( + US_DISABILITY_BENEFITS_OUTPUT_COLUMNS, +) +from populace.build.us_runtime.education_inputs import ( + US_EDUCATION_INPUTS_OUTPUT_COLUMNS, +) +from populace.build.us_runtime.eligibility_inputs import ( + US_ELIGIBILITY_INPUTS_OUTPUT_COLUMNS, +) +from populace.build.us_runtime.energy_subsidy import ( + US_ENERGY_SUBSIDY_OUTPUT_COLUMNS, +) +from populace.build.us_runtime.geography_ladder import US_GEOGRAPHY_LADDER_COLUMNS +from populace.build.us_runtime.housing_inputs import ( + US_HOUSING_HOUSEHOLD_OUTPUT_COLUMNS, + US_HOUSING_PERSON_OUTPUT_COLUMNS, + US_HOUSING_SPM_UNIT_OUTPUT_COLUMNS, +) +from populace.build.us_runtime.immigration import US_IMMIGRATION_OUTPUT_COLUMNS +from populace.build.us_runtime.medicare_take_up import ( + US_MEDICARE_TAKE_UP_OUTPUT_COLUMNS, +) +from populace.build.us_runtime.pregnancy import US_PREGNANCY_OUTPUT_COLUMN +from populace.build.us_runtime.prior_year_income import ( + US_PRIOR_YEAR_INCOME_OUTPUT_COLUMNS, +) +from populace.build.us_runtime.puf_capital_gains_tail import ( + PUF_CAPITAL_GAINS_TAIL_APPLIED_COLUMN, + PUF_CAPITAL_GAINS_TAIL_DONOR_AGI_BAND_COLUMN, + PUF_CAPITAL_GAINS_TAIL_DONOR_FILING_STATUS_COLUMN, + PUF_CAPITAL_GAINS_TAIL_DONOR_SOURCE_ID_COLUMN, + PUF_CAPITAL_GAINS_TAIL_DONOR_SYNTHETIC_COLUMN, + PUF_CAPITAL_GAINS_TAIL_PERSON_COLUMNS, + PUF_CAPITAL_GAINS_TAIL_TAX_UNIT_COLUMNS, + PUF_CAPITAL_GAINS_TAIL_TRANSFER_WEIGHT_COLUMN, +) +from populace.build.us_runtime.puf_support import ( + PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS, + PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS, +) +from populace.build.us_runtime.qbi_inputs import US_QBI_OUTPUT_COLUMNS +from populace.build.us_runtime.relationship_inputs import ( + US_RELATIONSHIP_INPUTS_OUTPUT_COLUMNS, +) +from populace.build.us_runtime.retirement_contributions import ( + US_RETIREMENT_CONTRIBUTION_OUTPUT_COLUMNS, +) +from populace.build.us_runtime.retirement_distributions import ( + US_RETIREMENT_DISTRIBUTION_OUTPUT_COLUMNS, +) +from populace.build.us_runtime.weeks_unemployed import ( + US_WEEKS_UNEMPLOYED_OUTPUT_COLUMNS, +) +from populace.build.us_runtime.wic_claim import US_WIC_CLAIM_OUTPUT_COLUMNS +from populace.build.us_runtime.workers_compensation import ( + US_WORKERS_COMPENSATION_OUTPUT_COLUMNS, +) +from populace.frame import Frame + +__all__ = [ + "PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES", + "assert_operator_free_source_frame", +] + +type OperatorOutputFamilies = Mapping[str, Mapping[str, frozenset[str]]] +type NativeInputReceipt = Mapping[str, Mapping[str, Any]] + +_US_ENTITIES = ( + "person", + "household", + "tax_unit", + "spm_unit", + "family", + "marital_unit", +) +_NATIVE_INPUT_RECEIPT_KEYS = frozenset( + { + "entity", + "missing_rows", + "observed_rows", + "provenance", + "source_columns", + "transformation", + } +) +_ACS_NATIVE_PROVENANCE = "acs_2024_1yr_native" +_ACS_NATIVE_INPUT_CONTRACTS: Mapping[ + str, + tuple[str, tuple[str, ...], str], +] = { + "age": ("person", ("AGEP",), "identity"), + "is_female": ("person", ("SEX",), "SEX == 2"), + "is_household_head": ("person", ("RELSHIPP",), "RELSHIPP == 20"), + "employment_income_before_lsr": ( + "person", + ("WAGP", "ADJINC"), + "WAGP * ADJINC / 1_000_000", + ), + "self_employment_income_before_lsr": ( + "person", + ("SEMP", "ADJINC"), + "SEMP * ADJINC / 1_000_000", + ), + "ssi_reported": ( + "person", + ("SSIP", "ADJINC"), + "SSIP * ADJINC / 1_000_000", + ), + "acs_social_security_income": ( + "person", + ("SSP", "ADJINC"), + "SSP * ADJINC / 1_000_000", + ), + "acs_retirement_income": ( + "person", + ("RETP", "ADJINC"), + "RETP * ADJINC / 1_000_000", + ), + "acs_interest_dividend_rental_income": ( + "person", + ("INTP", "ADJINC"), + "INTP * ADJINC / 1_000_000", + ), + "tenure_type": ("household", ("TEN",), "ACS TEN enum recode"), + "spm_unit_tenure_type": ( + "spm_unit", + ("TEN",), + "ACS TEN enum recode through SPM membership", + ), + "acs_monthly_contract_rent": ( + "household", + ("RNTP", "ADJHSG"), + "RNTP * ADJHSG / 1_000_000", + ), + "acs_monthly_gross_rent": ( + "household", + ("GRNTP", "ADJHSG"), + "GRNTP * ADJHSG / 1_000_000", + ), + "acs_annual_property_tax": ( + "household", + ("TAXAMT", "ADJHSG"), + "TAXAMT * ADJHSG / 1_000_000", + ), + "real_estate_taxes": ( + "person", + ("TAXAMT", "ADJHSG", "RELSHIPP"), + "TAXAMT * ADJHSG / 1_000_000; reference-person carry", + ), +} +_CAPITAL_GAINS_TAIL_PROVENANCE_COLUMNS = frozenset( + { + PUF_CAPITAL_GAINS_TAIL_APPLIED_COLUMN, + PUF_CAPITAL_GAINS_TAIL_DONOR_AGI_BAND_COLUMN, + PUF_CAPITAL_GAINS_TAIL_DONOR_FILING_STATUS_COLUMN, + PUF_CAPITAL_GAINS_TAIL_DONOR_SOURCE_ID_COLUMN, + PUF_CAPITAL_GAINS_TAIL_DONOR_SYNTHETIC_COLUMN, + PUF_CAPITAL_GAINS_TAIL_TRANSFER_WEIGHT_COLUMN, + } +) +_TAKE_UP_OPERATOR_OUTPUTS: Mapping[str, frozenset[str]] = { + "person": frozenset( + { + "takes_up_medicaid_if_eligible", + "takes_up_chip_if_eligible", + "takes_up_basic_health_program_if_eligible", + "takes_up_medicare_if_eligible", + "takes_up_ssi_if_eligible", + "takes_up_head_start_if_eligible", + "takes_up_early_head_start_if_eligible", + } + ), + "tax_unit": frozenset( + { + "takes_up_eitc", + "takes_up_dc_ptc", + "takes_up_aca_if_eligible", + } + ), + "spm_unit": frozenset( + { + "takes_up_snap_if_eligible", + "takes_up_tanf_if_eligible", + "takes_up_housing_assistance_if_eligible", + } + ), +} + + +PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES: OperatorOutputFamilies = { + "cps_carried": { + "person": frozenset(CPS_CARRIED_PERSON_INPUTS), + "spm_unit": frozenset(CPS_CARRIED_SPM_UNIT_INPUTS), + }, + "prior_year_income": { + "person": frozenset(US_PRIOR_YEAR_INCOME_OUTPUT_COLUMNS), + }, + "relationship_inputs": { + "person": frozenset(US_RELATIONSHIP_INPUTS_OUTPUT_COLUMNS), + }, + "medicare_take_up": { + "person": frozenset(US_MEDICARE_TAKE_UP_OUTPUT_COLUMNS), + }, + "housing_inputs": { + "person": frozenset(US_HOUSING_PERSON_OUTPUT_COLUMNS), + "household": frozenset(US_HOUSING_HOUSEHOLD_OUTPUT_COLUMNS), + "spm_unit": frozenset(US_HOUSING_SPM_UNIT_OUTPUT_COLUMNS), + }, + "eligibility_inputs": { + "person": frozenset(US_ELIGIBILITY_INPUTS_OUTPUT_COLUMNS), + }, + "pregnancy": { + "person": frozenset({US_PREGNANCY_OUTPUT_COLUMN}), + }, + "wic_claim": { + "person": frozenset(US_WIC_CLAIM_OUTPUT_COLUMNS), + }, + "child_support": { + "person": frozenset(US_CHILD_SUPPORT_OUTPUT_COLUMNS), + }, + "disability_benefits": { + "person": frozenset(US_DISABILITY_BENEFITS_OUTPUT_COLUMNS), + }, + "workers_compensation": { + "person": frozenset(US_WORKERS_COMPENSATION_OUTPUT_COLUMNS), + }, + "weeks_unemployed": { + "person": frozenset(US_WEEKS_UNEMPLOYED_OUTPUT_COLUMNS), + }, + "childcare": { + "spm_unit": frozenset(US_CHILDCARE_OUTPUT_COLUMNS), + }, + "energy_subsidy": { + "spm_unit": frozenset(US_ENERGY_SUBSIDY_OUTPUT_COLUMNS), + }, + "retirement_contributions": { + "person": frozenset(US_RETIREMENT_CONTRIBUTION_OUTPUT_COLUMNS), + }, + "retirement_distributions": { + "person": frozenset(US_RETIREMENT_DISTRIBUTION_OUTPUT_COLUMNS), + }, + "immigration": { + "person": frozenset(US_IMMIGRATION_OUTPUT_COLUMNS), + }, + "primary_puf_qrf": { + "person": frozenset(PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS), + "tax_unit": frozenset(PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS), + }, + "capital_gains_tail": { + "person": frozenset(PUF_CAPITAL_GAINS_TAIL_PERSON_COLUMNS), + "tax_unit": frozenset( + { + *PUF_CAPITAL_GAINS_TAIL_TAX_UNIT_COLUMNS, + *_CAPITAL_GAINS_TAIL_PROVENANCE_COLUMNS, + } + ), + }, + "capital_gain_distributions": { + "person": frozenset(ACS_DERIVED_TRANSFER_INPUTS), + }, + "qbi_reconciliation": { + "person": frozenset(US_QBI_OUTPUT_COLUMNS), + }, + "housing_assistance": { + "spm_unit": frozenset( + { + "receives_housing_assistance", + "takes_up_housing_assistance_if_eligible", + } + ), + }, + "adult_care": { + "person": frozenset(US_ADULT_CARE_OUTPUT_COLUMNS), + }, + "education_inputs": { + "person": frozenset(US_EDUCATION_INPUTS_OUTPUT_COLUMNS), + }, + "take_up": _TAKE_UP_OPERATOR_OUTPUTS, + "geography_assignment": { + "household": frozenset( + { + CONGRESSIONAL_DISTRICT_GEOID_COLUMN, + *US_GEOGRAPHY_LADDER_COLUMNS, + } + ), + }, + # These are not source inputs. They are included so a preassembled frame + # cannot smuggle formula evaluation or clone/assembly routing across the + # boundary under a family name outside the historical enrichment chain. + "formula_owned_aggregates": { + "person": frozenset({*CPS_CARRIED_FORMULA_OWNED_COLUMNS, "ssi"}), + }, + "support_provenance": { + entity: frozenset( + { + f"{entity}_spine_source_id", + f"{entity}_support_channel", + f"{entity}_support_clone_index", + f"{entity}_source_id", + } + ) + for entity in _US_ENTITIES + }, +} + + +def assert_operator_free_source_frame( + frame: Frame, + *, + label: str, + native_inputs: NativeInputReceipt | None = None, +) -> None: + """Require a source frame to be untouched by canonical US operators. + + ``native_inputs`` is only the flat receipt returned by + :func:`map_acs_native_inputs`; arbitrary allowlists are intentionally not + accepted. Every receipt entry is validated against the live frame before + it may account for a canonical-looking native ACS column. + """ + + if not isinstance(frame, Frame): + raise TypeError(f"{label} must be a Frame, got {type(frame).__name__}.") + if not isinstance(label, str) or not label.strip(): + raise ValueError("operator-boundary label must be a non-empty string.") + allowed = _validated_native_inputs(frame, native_inputs, label=label) + violations: list[str] = [] + for family, by_entity in PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES.items(): + for entity, columns in by_entity.items(): + if entity not in frame.entities: + continue + present = sorted( + column + for column in columns.intersection(frame.table(entity).columns) + if (entity, column) not in allowed + ) + if present: + violations.append(f"{family}:{entity}={present}") + if violations: + raise ValueError( + f"{label} must be operator-free before spine assembly; canonical " + f"operator output column family violation(s): {'; '.join(violations)}." + ) + + +def _validated_native_inputs( + frame: Frame, + native_inputs: NativeInputReceipt | None, + *, + label: str, +) -> frozenset[tuple[str, str]]: + if native_inputs is None: + return frozenset() + if not isinstance(native_inputs, Mapping): + raise ValueError( + f"{label} native_inputs must be the flat ACS native-mapping receipt." + ) + allowed: set[tuple[str, str]] = set() + all_columns = { + column + for entity in frame.entities + for column in frame.table(entity).columns + } + for output, raw_receipt in native_inputs.items(): + if not isinstance(output, str) or not output: + raise ValueError(f"{label} native_inputs output names must be strings.") + contract = _ACS_NATIVE_INPUT_CONTRACTS.get(output) + if contract is None: + raise ValueError( + f"{label} native_inputs[{output!r}] is not a declared ACS native " + "mapping output." + ) + if not isinstance(raw_receipt, Mapping) or frozenset(raw_receipt) != ( + _NATIVE_INPUT_RECEIPT_KEYS + ): + raise ValueError( + f"{label} native_inputs[{output!r}] is not an exact ACS native " + "mapping receipt." + ) + entity = raw_receipt["entity"] + source_columns = raw_receipt["source_columns"] + transformation = raw_receipt["transformation"] + provenance = raw_receipt["provenance"] + observed_rows = raw_receipt["observed_rows"] + missing_rows = raw_receipt["missing_rows"] + expected_entity, expected_sources, expected_transformation = contract + if entity != expected_entity: + raise ValueError( + f"{label} native_inputs[{output!r}].entity must match the declared " + f"ACS mapping entity {expected_entity!r}." + ) + if ( + not isinstance(source_columns, list) + or tuple(source_columns) != expected_sources + or any(column not in all_columns for column in source_columns) + ): + raise ValueError( + f"{label} native_inputs[{output!r}].source_columns must exactly " + f"match the declared ACS mapping {list(expected_sources)!r} and " + "be present raw frame columns." + ) + if transformation != expected_transformation: + raise ValueError( + f"{label} native_inputs[{output!r}].transformation must match the " + "declared ACS mapping." + ) + if provenance != _ACS_NATIVE_PROVENANCE: + raise ValueError( + f"{label} native_inputs[{output!r}].provenance must be " + f"{_ACS_NATIVE_PROVENANCE!r}." + ) + _require_row_count(observed_rows, label=label, output=output, field="observed") + _require_row_count(missing_rows, label=label, output=output, field="missing") + table = frame.table(entity) + if output not in table: + raise ValueError( + f"{label} native_inputs[{output!r}] does not name a column on " + f"entity {entity!r}." + ) + actual_missing = int(pd.isna(table[output]).sum()) + actual_observed = len(table) - actual_missing + if observed_rows != actual_observed or missing_rows != actual_missing: + raise ValueError( + f"{label} native_inputs[{output!r}] row counts do not match the " + "live frame." + ) + allowed.add((entity, output)) + return frozenset(allowed) + + +def _require_row_count( + value: object, + *, + label: str, + output: str, + field: str, +) -> None: + if not isinstance(value, int) or isinstance(value, bool) or value < 0: + raise ValueError( + f"{label} native_inputs[{output!r}].{field}_rows must be a " + "nonnegative integer." + ) diff --git a/packages/populace-build/src/populace/build/us_runtime/spine_agreement.py b/packages/populace-build/src/populace/build/us_runtime/spine_agreement.py index c13bf4e9..cd0d0710 100644 --- a/packages/populace-build/src/populace/build/us_runtime/spine_agreement.py +++ b/packages/populace-build/src/populace/build/us_runtime/spine_agreement.py @@ -14,6 +14,11 @@ * at weighted conditional quantiles 10, 25, 50, 75, and 90 percent, the maximum symmetric relative distance must be no greater than ``0.25``. +Categorical surfaces instead compare the complete weighted category +distribution by total-variation distance, which must be no greater than +``0.25``. Declared joint categorical groups are additionally compared as +tuples, so equal marginals cannot conceal invalid cross-column combinations. + The symmetric quantile distance at one quantile is ``2 * abs(left - right) / (abs(left) + abs(right))``. It is zero when both quantiles are zero and at most two otherwise. Quantiles use positive-weight, @@ -48,6 +53,7 @@ from populace.build.us_runtime.take_up_contract import load_take_up_contract __all__ = [ + "DEFAULT_CATEGORICAL_TOTAL_VARIATION_TOLERANCE", "DEFAULT_INCIDENCE_RATIO_BOUNDS", "DEFAULT_QUANTILE_ENVELOPE_TOLERANCE", "DEFAULT_SPINE_AGREEMENT_QUANTILES", @@ -62,6 +68,7 @@ DEFAULT_INCIDENCE_RATIO_BOUNDS = (0.8, 1.25) DEFAULT_SPINE_AGREEMENT_QUANTILES = (0.10, 0.25, 0.50, 0.75, 0.90) DEFAULT_QUANTILE_ENVELOPE_TOLERANCE = 0.25 +DEFAULT_CATEGORICAL_TOTAL_VARIATION_TOLERANCE = 0.25 _BATCH_SEPARATOR = "__batch_" _DERIVED_FAMILY = "derived_transfer" _SIMULATED_OUTPUT_FAMILY = "simulated_output" @@ -70,6 +77,9 @@ _SIMULATED_OUTPUTS: Mapping[str, tuple[str, ...]] = { "person": ("ssi",), } +_JOINT_CATEGORICAL_TARGET_GROUPS: tuple[tuple[str, ...], ...] = ( + ("ssn_card_type", "immigration_status_str"), +) class _ResolvedWeights(Protocol): @@ -95,6 +105,7 @@ class SpineAgreementSpec: entity: str family: str columns: tuple[str, ...] + joint_categorical_groups: tuple[tuple[str, ...], ...] = () def __post_init__(self) -> None: if not isinstance(self.entity, str) or not self.entity.strip(): @@ -127,6 +138,48 @@ def __post_init__(self) -> None: f"SpineAgreementSpec {self.entity}/{self.family} repeats columns " f"{duplicate_columns}." ) + if not isinstance(self.joint_categorical_groups, tuple) or any( + not isinstance(group, tuple) + for group in self.joint_categorical_groups + ): + raise ValueError( + "SpineAgreementSpec.joint_categorical_groups must be an " + "immutable tuple of tuples." + ) + invalid_groups = [ + group + for group in self.joint_categorical_groups + if len(group) < 2 + or any( + not isinstance(column, str) or not column.strip() + for column in group + ) + or len(set(group)) != len(group) + or not set(group).issubset(self.columns) + ] + if invalid_groups: + raise ValueError( + f"SpineAgreementSpec {self.entity}/{self.family} has invalid " + f"joint categorical group(s): {invalid_groups}." + ) + grouped_columns = [ + column + for group in self.joint_categorical_groups + for column in group + ] + repeated_group_columns = sorted( + { + column + for column in grouped_columns + if grouped_columns.count(column) > 1 + } + ) + if repeated_group_columns: + raise ValueError( + f"SpineAgreementSpec {self.entity}/{self.family} repeats columns " + "across joint categorical groups: " + f"{repeated_group_columns}." + ) def normalize_transfer_family_name(family: str) -> str: @@ -171,6 +224,11 @@ def default_spine_agreement_registry( entity=entity, family=family, columns=tuple(dict.fromkeys(columns)), + joint_categorical_groups=tuple( + group + for group in _JOINT_CATEGORICAL_TARGET_GROUPS + if set(group).issubset(columns) + ), ) for (entity, family), columns in sorted(normalized.items()) ) @@ -391,25 +449,66 @@ def spine_agreement_gate( checked_columns = 0 checked_pairs = 0 tested_pairs = 0 + checked_joint_categorical_groups = 0 for spec in specs: context = contexts.get(spec.entity) if context is None: continue table, weights, observed_channels = context channel_values = table[f"{spec.entity}_support_channel"].astype(str).to_numpy() + comparison_channels = ( + expected_source_channels + if len(expected_source_channels) >= 2 + else observed_channels + ) for column in spec.columns: label = f"{spec.entity}/{spec.family}/{column}" if column not in table: failures.append(f"{label}: registered column is absent from the frame.") continue series = table[column] + relevant = weights > 0.0 if not (is_numeric_dtype(series) or is_bool_dtype(series)): - failures.append( - f"{label}: dtype {series.dtype} is not numeric or boolean." + invalid_values = relevant & series.isna().to_numpy(dtype=bool) + if invalid_values.any(): + failures.append( + f"{label}: {int(invalid_values.sum())} positive-weight " + "categorical value(s) are missing." + ) + continue + category_values = series.to_numpy(dtype=object) + unhashable = sum( + 1 + for value, included in zip( + category_values, + relevant, + strict=True, + ) + if included and not _is_hashable(value) + ) + if unhashable: + failures.append( + f"{label}: {unhashable} positive-weight categorical " + "value(s) are not hashable." + ) + continue + checked_columns += 1 + pair_count, tested_count = _record_categorical_comparisons( + label=label, + values=category_values, + weights=weights, + channel_values=channel_values, + observed_channels=observed_channels, + comparison_channels=comparison_channels, + failures=failures, + comparisons=comparisons, + untestable_comparisons=untestable_comparisons, ) + checked_pairs += pair_count + tested_pairs += tested_count continue + values = series.to_numpy(dtype=np.float64, na_value=np.nan) - relevant = weights > 0.0 invalid_values = relevant & ~np.isfinite(values) if invalid_values.any(): failures.append( @@ -420,11 +519,6 @@ def spine_agreement_gate( checked_columns += 1 lower, upper = DEFAULT_INCIDENCE_RATIO_BOUNDS - comparison_channels = ( - expected_source_channels - if len(expected_source_channels) >= 2 - else observed_channels - ) for left_channel, right_channel in itertools.combinations( comparison_channels, 2 ): @@ -493,6 +587,62 @@ def spine_agreement_gate( f"{DEFAULT_QUANTILE_ENVELOPE_TOLERANCE:.6g}." ) + for group in spec.joint_categorical_groups: + if any(column not in table for column in group): + continue + group_series = [table[column] for column in group] + if any( + is_numeric_dtype(series) or is_bool_dtype(series) + for series in group_series + ): + failures.append( + f"{spec.entity}/{spec.family}/joint[{','.join(group)}]: " + "registered joint categorical columns must all be categorical." + ) + continue + relevant = weights > 0.0 + invalid_values = relevant & np.logical_or.reduce( + [series.isna().to_numpy(dtype=bool) for series in group_series] + ) + if invalid_values.any(): + continue + category_values = np.empty(len(table), dtype=object) + category_values[:] = list( + zip( + *[ + series.to_numpy(dtype=object) + for series in group_series + ], + strict=True, + ) + ) + if any( + included and not _is_hashable(value) + for value, included in zip( + category_values, + relevant, + strict=True, + ) + ): + continue + checked_joint_categorical_groups += 1 + pair_count, tested_count = _record_categorical_comparisons( + label=( + f"{spec.entity}/{spec.family}/" + f"joint[{','.join(group)}]" + ), + values=category_values, + weights=weights, + channel_values=channel_values, + observed_channels=observed_channels, + comparison_channels=comparison_channels, + failures=failures, + comparisons=comparisons, + untestable_comparisons=untestable_comparisons, + ) + checked_pairs += pair_count + tested_pairs += tested_count + return GateResult( name=_GATE_NAME, passed=not failures, @@ -507,13 +657,26 @@ def spine_agreement_gate( "quantile_distance": ( "max_q 2*abs(left_q-right_q)/(abs(left_q)+abs(right_q))" ), + "categorical_distribution": ( + "resolved-weight category shares, including registered " + "joint tuples" + ), + "categorical_distance": ( + "0.5 * sum_category abs(left_share-right_share)" + ), }, "tolerances": { "incidence_ratio_bounds": list(DEFAULT_INCIDENCE_RATIO_BOUNDS), "max_quantile_envelope_distance": (DEFAULT_QUANTILE_ENVELOPE_TOLERANCE), + "max_categorical_total_variation_distance": ( + DEFAULT_CATEGORICAL_TOTAL_VARIATION_TOLERANCE + ), }, "registered_families": len(specs), "checked_columns": checked_columns, + "checked_joint_categorical_groups": ( + checked_joint_categorical_groups + ), "checked_spine_pairs": checked_pairs, "tested_spine_pairs": tested_pairs, "untestable_comparisons": sorted(untestable_comparisons), @@ -603,6 +766,127 @@ def _normalize_target_families( return normalized +def _record_categorical_comparisons( + *, + label: str, + values: np.ndarray, + weights: np.ndarray, + channel_values: np.ndarray, + observed_channels: tuple[str, ...], + comparison_channels: tuple[str, ...], + failures: list[str], + comparisons: dict[str, object], + untestable_comparisons: list[str], +) -> tuple[int, int]: + """Record fixed weighted total-variation checks for one category surface.""" + + checked_pairs = 0 + tested_pairs = 0 + for left_channel, right_channel in itertools.combinations( + comparison_channels, + 2, + ): + checked_pairs += 1 + comparison_key = f"{label}/{left_channel}_vs_{right_channel}" + missing_pair_channels = sorted( + {left_channel, right_channel} - set(observed_channels) + ) + if missing_pair_channels: + comparisons[comparison_key] = { + "status": "untestable_missing_source_spine", + "missing_source_spines": missing_pair_channels, + } + untestable_comparisons.append(comparison_key) + continue + + left = channel_values == left_channel + right = channel_values == right_channel + left_distribution = _weighted_category_distribution( + values[left], + weights[left], + ) + right_distribution = _weighted_category_distribution( + values[right], + weights[right], + ) + distance = _categorical_total_variation_distance( + left_distribution, + right_distribution, + ) + tested_pairs += 1 + comparisons[comparison_key] = { + "status": "tested", + "left_category_shares": _manifest_category_shares( + left_distribution + ), + "right_category_shares": _manifest_category_shares( + right_distribution + ), + "categorical_total_variation_distance": distance, + } + if distance > DEFAULT_CATEGORICAL_TOTAL_VARIATION_TOLERANCE: + failures.append( + f"{comparison_key}: weighted categorical total-variation " + f"distance {distance:.6g} exceeds " + f"{DEFAULT_CATEGORICAL_TOTAL_VARIATION_TOLERANCE:.6g}." + ) + return checked_pairs, tested_pairs + + +def _weighted_category_distribution( + values: np.ndarray, + weights: np.ndarray, +) -> dict[object, float]: + total_weight = float(weights.sum()) + distribution: dict[object, float] = {} + for value, weight in zip(values, weights, strict=True): + if weight <= 0.0: + continue + distribution[value] = distribution.get(value, 0.0) + float(weight) + return { + category: category_weight / total_weight + for category, category_weight in distribution.items() + } + + +def _categorical_total_variation_distance( + left: Mapping[object, float], + right: Mapping[object, float], +) -> float: + categories = set(left) | set(right) + return 0.5 * sum( + abs(left.get(category, 0.0) - right.get(category, 0.0)) + for category in categories + ) + + +def _manifest_category_shares( + distribution: Mapping[object, float], +) -> dict[str, float]: + return { + _manifest_category(category): distribution[category] + for category in sorted(distribution, key=_category_sort_key) + } + + +def _manifest_category(value: object) -> str: + value_type = type(value) + return f"{value_type.__module__}.{value_type.__qualname__}:{value!r}" + + +def _category_sort_key(value: object) -> tuple[str, str, str]: + value_type = type(value) + return (value_type.__module__, value_type.__qualname__, repr(value)) + + +def _is_hashable(value: object) -> bool: + try: + hash(value) + except TypeError: + return False + return True + + def _weighted_incidence(values: np.ndarray, weights: np.ndarray) -> float: return float(weights[values != 0.0].sum() / weights.sum()) diff --git a/packages/populace-build/tests/test_us_asec_checkpoint.py b/packages/populace-build/tests/test_us_asec_checkpoint.py index aa5e0e53..b76536d8 100644 --- a/packages/populace-build/tests/test_us_asec_checkpoint.py +++ b/packages/populace-build/tests/test_us_asec_checkpoint.py @@ -12,7 +12,17 @@ OUTER_STAGE_CONTEXT_SCHEMA_VERSION, frame_identity, ) -from populace.build.us_runtime import load_asec_pre_clone_checkpoint +from populace.build.us_runtime import ( + ASEC_RAW_STAGE_ARTIFACT_KIND, + ASEC_RAW_STAGE_OPERATOR_STATUS, + ASEC_RAW_STAGE_SCHEMA_VERSION, + ASEC_RAW_STAGE_STAGE, + PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES, + assert_operator_free_source_frame, + load_asec_pre_clone_checkpoint, + load_asec_raw_stage_checkpoint, + load_take_up_contract, +) from populace.frame import US_SCHEMA, EntitySchema, Frame, WeightKind, Weights _OUTER_STAGE_ARTIFACT_KIND = "populace_outer_stage_frame" @@ -22,6 +32,7 @@ def _us_frame( *, id_offset: int = 0, household_weights: tuple[float, float] = (2.0, 3.0), + include_age: bool = True, person_weights: bool = False, ) -> Frame: ids = np.asarray([1, 2], dtype=np.int64) + id_offset @@ -33,9 +44,10 @@ def _us_frame( "person_spm_unit_id": ids + 20, "person_family_id": ids + 30, "person_marital_unit_id": ids + 40, - "age": np.asarray([30, 50], dtype=np.int16), } ) + if include_age: + person["age"] = np.asarray([30, 50], dtype=np.int16) tables = { "person": person, "household": pd.DataFrame({"household_id": ids}), @@ -99,6 +111,70 @@ def _binding(frame: Frame) -> dict[str, object]: } +def _raw_binding(frame: Frame) -> dict[str, object]: + pin = { + "income_year": 2022, + "locator": "https://example.test/asec.zip", + "member": "pppub.csv", + "member_sha256": "b" * 64, + "sha256": "a" * 64, + } + return { + "artifact_kind": ASEC_RAW_STAGE_ARTIFACT_KIND, + "identity": frame_identity(frame).to_payload(), + "operator_status": ASEC_RAW_STAGE_OPERATOR_STATUS, + "pipeline_sha256": "c" * 64, + "raw_source_mappings": { + column: { + "audit": {"rows": 2}, + "column": column, + "entity": "person", + "join_keys": ["source_year", "PERIDNUM"], + "operation": "exact_source_join", + "source_pins": [pin], + } + for column in ("ED_VAL", "LKWEEKS") + }, + "schema_version": ASEC_RAW_STAGE_SCHEMA_VERSION, + "source_construction_identity": frame_identity(frame).to_payload(), + "source_receipt": { + "kind": "pooled_asec", + "sources": [ + { + "max_households": None, + "path": "/raw/asec_2022.h5", + "sha256": "d" * 64, + "share": 1.0, + "year": 2022, + } + ], + "target_year": 2022, + }, + "stage": ASEC_RAW_STAGE_STAGE, + } + + +def _raw_us_frame(*, id_offset: int = 0) -> Frame: + source = _us_frame(id_offset=id_offset, include_age=False) + tables = {entity: source.table(entity).copy() for entity in source.entities} + tables["person"]["source_year"] = [2022, 2023] + tables["person"]["PERIDNUM"] = [ + "0000000000000000000001", + "0000000000000000000002", + ] + tables["person"]["ED_VAL"] = [0.0, 500.0] + tables["person"]["LKWEEKS"] = [-1, 12] + return Frame( + tables, + source.schema, + { + entity: source.weights_for(entity) + for entity in source.weighted_entities + }, + source.strata, + ) + + def _write_checkpoint( path: Path, frame: Frame, @@ -149,6 +225,250 @@ def test_rejects_wrong_outer_stage_binding( load_asec_pre_clone_checkpoint(path) +def test_loads_operator_untouched_raw_stage_checkpoint(tmp_path: Path) -> None: + path = tmp_path / "asec_raw_stage.checkpoint.h5" + source = _raw_us_frame() + metadata = _raw_binding(source) + _write_checkpoint(path, source, metadata=metadata) + + frame, loaded_metadata = load_asec_raw_stage_checkpoint(path) + + assert frame_identity(frame) == frame_identity(source) + assert loaded_metadata == metadata + assert loaded_metadata["artifact_kind"] == ASEC_RAW_STAGE_ARTIFACT_KIND + assert loaded_metadata["stage"] == ASEC_RAW_STAGE_STAGE + assert loaded_metadata["operator_status"] == ASEC_RAW_STAGE_OPERATOR_STATUS + + +@pytest.mark.parametrize("column", ("ED_VAL", "LKWEEKS", "PERIDNUM", "source_year")) +def test_raw_loader_rejects_missing_input_complete_source_column( + tmp_path: Path, + column: str, +) -> None: + source = _raw_us_frame() + tables = {entity: source.table(entity).copy() for entity in source.entities} + tables["person"] = tables["person"].drop(columns=[column]) + incomplete = Frame( + tables, + source.schema, + { + entity: source.weights_for(entity) + for entity in source.weighted_entities + }, + source.strata, + ) + path = tmp_path / f"raw-missing-{column}.checkpoint.h5" + _write_checkpoint(path, incomplete, metadata=_raw_binding(incomplete)) + + with pytest.raises(ValueError, match=rf"input-complete.*{column}"): + load_asec_raw_stage_checkpoint(path) + + +@pytest.mark.parametrize( + ("column", "values", "message"), + ( + ("source_year", [2022, np.nan], "source_year must be complete"), + ("PERIDNUM", ["0000000000000000000001", ""], "PERIDNUM must be complete"), + ("ED_VAL", [0.0, np.nan], "ED_VAL must be complete"), + ("LKWEEKS", [-1, 53], "LKWEEKS must be complete"), + ), +) +def test_raw_loader_rejects_invalid_input_complete_source_values( + tmp_path: Path, + column: str, + values: list[object], + message: str, +) -> None: + source = _raw_us_frame() + tables = {entity: source.table(entity).copy() for entity in source.entities} + tables["person"][column] = values + invalid = Frame( + tables, + source.schema, + { + entity: source.weights_for(entity) + for entity in source.weighted_entities + }, + source.strata, + ) + path = tmp_path / f"raw-invalid-{column}.checkpoint.h5" + _write_checkpoint(path, invalid, metadata=_raw_binding(invalid)) + + with pytest.raises(ValueError, match=message): + load_asec_raw_stage_checkpoint(path) + + +@pytest.mark.parametrize( + ("field", "value", "message"), + ( + ("artifact_kind", _OUTER_STAGE_ARTIFACT_KIND, "not a dedicated raw-stage"), + ("schema_version", 999, "unsupported raw-stage schema version"), + ("stage", "pre_clone_enrichment", "must be bound to stage"), + ("operator_status", "operator_enriched", "must declare operator_status"), + ("pipeline_sha256", "not-a-digest", "lowercase SHA-256 digest"), + ), +) +def test_raw_loader_rejects_wrong_artifact_binding( + tmp_path: Path, + field: str, + value: object, + message: str, +) -> None: + path = tmp_path / f"raw-wrong-{field}.checkpoint.h5" + frame = _raw_us_frame() + metadata = _raw_binding(frame) + metadata[field] = value + _write_checkpoint(path, frame, metadata=metadata) + + with pytest.raises(ValueError, match=message): + load_asec_raw_stage_checkpoint(path) + + +def test_raw_loader_rejects_legacy_enriched_checkpoint(tmp_path: Path) -> None: + path = tmp_path / "001_pre_clone_enrichment.frame.h5" + frame = _us_frame() + _write_checkpoint(path, frame, metadata=_binding(frame)) + + with pytest.raises(ValueError, match="incomplete raw-stage artifact binding"): + load_asec_raw_stage_checkpoint(path) + + +def test_raw_loader_rejects_identity_not_bound_to_frame(tmp_path: Path) -> None: + path = tmp_path / "raw-wrong-identity.checkpoint.h5" + frame = _raw_us_frame() + metadata = _raw_binding(frame) + metadata["identity"] = frame_identity(_raw_us_frame(id_offset=100)).to_payload() + _write_checkpoint(path, frame, metadata=metadata) + + with pytest.raises(ValueError, match="Frame identity changed"): + load_asec_raw_stage_checkpoint(path) + + +_OPERATOR_OUTPUT_CASES = tuple( + (family, entity, column) + for family, by_entity in PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES.items() + for entity, columns in by_entity.items() + for column in sorted(columns) +) + + +def test_operator_boundary_enumerates_full_take_up_contract() -> None: + expected: dict[str, set[str]] = {} + for program in load_take_up_contract().programs: + expected.setdefault(program.entity, set()).add(program.variable) + + assert PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES["take_up"] == { + entity: frozenset(columns) for entity, columns in expected.items() + } + + +def test_operator_boundary_accepts_only_receipted_acs_native_exception() -> None: + source = _us_frame(include_age=False) + tables = {entity: source.table(entity).copy() for entity in source.entities} + tables["person"]["AGEP"] = [30, 50] + tables["person"]["age"] = [30, 50] + acs = Frame( + tables, + source.schema, + { + entity: source.weights_for(entity) + for entity in source.weighted_entities + }, + source.strata, + ) + receipt = { + "age": { + "entity": "person", + "source_columns": ["AGEP"], + "transformation": "identity", + "provenance": "acs_2024_1yr_native", + "observed_rows": 2, + "missing_rows": 0, + } + } + + assert_operator_free_source_frame( + acs, + label="ACS fixture", + native_inputs=receipt, + ) + with pytest.raises(ValueError, match="canonical operator output"): + assert_operator_free_source_frame(acs, label="unreceipted ACS fixture") + + malformed = {"age": {**receipt["age"], "provenance": "fixture_allowlist"}} + with pytest.raises(ValueError, match="provenance"): + assert_operator_free_source_frame( + acs, + label="malformed ACS fixture", + native_inputs=malformed, + ) + + +def test_operator_boundary_rejects_forged_native_receipt_for_operator_output() -> None: + source = _us_frame(include_age=False) + tables = {entity: source.table(entity).copy() for entity in source.entities} + tables["person"]["AGEP"] = [30, 50] + tables["person"]["would_claim_wic"] = [True, False] + forged = Frame( + tables, + source.schema, + { + entity: source.weights_for(entity) + for entity in source.weighted_entities + }, + source.strata, + ) + forged_receipt = { + "would_claim_wic": { + "entity": "person", + "source_columns": ["AGEP"], + "transformation": "identity", + "provenance": "acs_2024_1yr_native", + "observed_rows": 2, + "missing_rows": 0, + } + } + + with pytest.raises(ValueError, match="not a declared ACS native mapping"): + assert_operator_free_source_frame( + forged, + label="forged ACS fixture", + native_inputs=forged_receipt, + ) + + +@pytest.mark.parametrize( + ("family", "entity", "column"), + _OPERATOR_OUTPUT_CASES, +) +def test_raw_loader_rejects_every_registered_operator_output( + tmp_path: Path, + family: str, + entity: str, + column: str, +) -> None: + source = _raw_us_frame() + tables = { + table_entity: source.table(table_entity).copy() + for table_entity in source.entities + } + tables[entity][column] = 0 + contaminated = Frame( + tables, + source.schema, + { + weighted_entity: source.weights_for(weighted_entity) + for weighted_entity in source.weighted_entities + }, + source.strata, + ) + path = tmp_path / f"{family}-{entity}-{column}.checkpoint.h5" + _write_checkpoint(path, contaminated, metadata=_raw_binding(contaminated)) + + with pytest.raises(ValueError, match=rf"{family}:{entity}"): + load_asec_raw_stage_checkpoint(path) + + def test_rejects_incomplete_outer_stage_binding(tmp_path: Path) -> None: path = tmp_path / "missing-stage.frame.h5" frame = _us_frame() diff --git a/packages/populace-build/tests/test_us_multispine_pool.py b/packages/populace-build/tests/test_us_multispine_pool.py index b5c952fa..a8a2cb70 100644 --- a/packages/populace-build/tests/test_us_multispine_pool.py +++ b/packages/populace-build/tests/test_us_multispine_pool.py @@ -8,24 +8,34 @@ import pytest from populace.build.gates import GateResult +from populace.build.us_runtime import multispine_pool as multispine_pool_module from populace.build.us_runtime.acs_transfer import ( declared_acs_transfer_target_families, ) from populace.build.us_runtime.multispine_pool import ( POOL_OPERATOR_ORDER, + POOL_SOURCE_OPERATOR_ORDER, + POOL_SPINE_AGREEMENT_REGISTRY, PoolStageOutput, _complete_schedule_d_input, materialize_multispine_agreement_outputs, pool_transfer_target_families, + prepare_multispine_puf_predictors, run_multispine_pool_path, seed_multispine_pool_inputs, ) +from populace.build.us_runtime.operator_boundary import ( + PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES, +) from populace.build.us_runtime.puf_support import ( PUF_SUPPORT_MAX_CLONE_SAFE_SOURCE_ID, + clone_us_frame_for_puf_support, ) from populace.build.us_runtime.spine_agreement import ( SpineAgreementSpec, + default_spine_agreement_registry, spine_agreement_gate, + validate_spine_agreement_registry, ) from populace.build.us_runtime.spine_assembly import assemble_spines from populace.build.us_runtime.support_provenance import ( @@ -35,6 +45,29 @@ from populace.build.us_runtime.take_up_contract import load_take_up_contract from populace.frame import US_SCHEMA, Frame, WeightKind, Weights +_EXPECTED_POOL_SOURCE_OPERATOR_ORDER = ( + "derive_us_cps_carried_inputs", + "with_us_prior_year_income_inputs", + "with_us_relationship_inputs", + "with_us_medicare_take_up_input", + "with_us_housing_inputs", + "with_us_eligibility_inputs", + "with_us_pregnancy_inputs", + "with_us_wic_claim_input", + "impute_us_housing_assistance_to_puf_support", + "with_us_child_support_inputs", + "with_us_disability_benefits", + "with_us_workers_compensation", + "with_us_weeks_unemployed", + "with_us_childcare_inputs", + "with_us_adult_care_inputs", + "with_us_energy_subsidy_input", + "with_us_retirement_contribution_inputs", + "with_us_retirement_distribution_inputs", + "with_us_immigration_inputs", + "with_us_education_inputs", +) + def _source_frame(*, offset: float = 0.0) -> Frame: ids = np.asarray([1, 2], dtype=np.int64) @@ -208,6 +241,7 @@ def simulated(person: pd.DataFrame) -> None: assert result.agreement_gate.details["tolerances"] == { "incidence_ratio_bounds": [0.8, 1.25], "max_quantile_envelope_distance": 0.25, + "max_categorical_total_variation_distance": 0.25, } assert order == ["impute", "derive", "seed", "simulate"] @@ -261,8 +295,315 @@ def unreachable(_frame: Frame) -> PoolStageOutput: ) -def test_pool_transfer_plan_is_the_fixed_declared_qrf_surface() -> None: - assert pool_transfer_target_families() == declared_acs_transfer_target_families() +def test_pool_transfer_plan_extends_legacy_without_duplicates() -> None: + legacy = declared_acs_transfer_target_families() + pool = pool_transfer_target_families() + + for entity, families in legacy.items(): + for family, columns in families.items(): + assert pool[entity][family] == columns + + owners: dict[str, tuple[str, str]] = {} + for entity, families in pool.items(): + for family, columns in families.items(): + for column in columns: + assert column not in owners, ( + f"{column} is duplicated by {owners[column]} and " + f"{(entity, family)}" + ) + owners[column] = (entity, family) + + assert owners["takes_up_medicare_if_eligible"] == ( + "person", + "source_operator_medicare_take_up", + ) + assert owners["receives_housing_assistance"] == ( + "spm_unit", + "source_operator_housing_inputs", + ) + assert owners["immigration_status_str"] == ( + "person", + "source_operator_immigration", + ) + + +def test_pool_agreement_registry_exactly_covers_expanded_pool_charter() -> None: + target_families = pool_transfer_target_families() + + assert POOL_SPINE_AGREEMENT_REGISTRY == default_spine_agreement_registry( + target_families + ) + assert ( + validate_spine_agreement_registry( + POOL_SPINE_AGREEMENT_REGISTRY, + target_families=target_families, + ) + == POOL_SPINE_AGREEMENT_REGISTRY + ) + registered = { + (spec.entity, column) + for spec in POOL_SPINE_AGREEMENT_REGISTRY + for column in spec.columns + } + transferred = { + (entity, column) + for entity, families in target_families.items() + for columns in families.values() + for column in columns + } + take_up = { + (program.entity, program.variable) + for program in load_take_up_contract().programs + } + assert transferred | take_up | {("person", "ssi")} <= registered + + immigration_spec = next( + spec + for spec in POOL_SPINE_AGREEMENT_REGISTRY + if (spec.entity, spec.family) + == ("person", "source_operator_immigration") + ) + assert immigration_spec.columns == ( + "immigration_status_str", + "ssn_card_type", + ) + assert immigration_spec.joint_categorical_groups == ( + ("ssn_card_type", "immigration_status_str"), + ) + + +def test_production_path_passes_fixed_pool_registry_to_gate( + monkeypatch: pytest.MonkeyPatch, +) -> None: + captured: list[object] = [] + + def no_op(frame: Frame) -> PoolStageOutput: + return PoolStageOutput(frame) + + def gate( + _frame: Frame, + *, + registry: object, + ) -> GateResult: + captured.append(registry) + return GateResult("fixture", True) + + monkeypatch.setattr(multispine_pool_module, "spine_agreement_gate", gate) + + result = run_multispine_pool_path( + _source_frame(), + _source_frame(), + impute=no_op, + derive=no_op, + seed=no_op, + simulate=no_op, + ) + + assert result.simulation_ready + assert captured == [POOL_SPINE_AGREEMENT_REGISTRY] + + +def test_pool_source_operator_order_is_the_full_legacy_chain() -> None: + assert POOL_SOURCE_OPERATOR_ORDER == _EXPECTED_POOL_SOURCE_OPERATOR_ORDER + + +def test_every_source_operator_output_has_a_pool_owner() -> None: + transferred = { + column + for families in pool_transfer_target_families().values() + for columns in families.values() + for column in columns + } + native = { + column + for columns in multispine_pool_module._POOL_NATIVE_COMPLETE_OUTPUTS.values() + for column in columns + } + formula_owned = { + column + for columns in multispine_pool_module._FORMULA_OWNED_SOURCE_OUTPUTS.values() + for column in columns + } + seeded = { + program.variable + for program in load_take_up_contract().programs + if program.is_seeded + } + + unowned: list[str] = [] + for operator_name in POOL_SOURCE_OPERATOR_ORDER: + family = multispine_pool_module._SOURCE_OPERATOR_FAMILIES[operator_name] + for columns in PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES[family].values(): + for column in columns: + if column not in transferred | native | formula_owned | seeded: + unowned.append(f"{family}.{column}") + assert not unowned + + +def test_source_operator_chain_is_availability_aware_and_source_blind( + monkeypatch: pytest.MonkeyPatch, +) -> None: + asec = _source_frame() + asec_tables = {entity: asec.table(entity).copy() for entity in asec.entities} + asec_tables["person"]["PERIDNUM"] = ["asec-1", "asec-2"] + asec = Frame( + asec_tables, + asec.schema, + {"household": asec.weights_for("household")}, + asec.strata, + ) + acs = _source_frame(offset=100.0) + acs_tables = {entity: acs.table(entity).copy() for entity in acs.entities} + first_output = "fixture_source_output_00" + acs_tables["person"][first_output] = [900.0, 901.0] + acs = Frame( + acs_tables, + acs.schema, + {"household": acs.weights_for("household")}, + acs.strata, + ) + assembled = assemble_spines( + {"asec": asec, "acs": acs}, + household_mass_shares={"asec": 0.5, "acs": 0.5}, + ) + cloned = clone_us_frame_for_puf_support(assembled) + + calls: list[str] = [] + output_families: dict[str, dict[str, frozenset[str]]] = {} + operators: dict[str, Callable[[Frame], Frame]] = {} + for index, operator_name in enumerate(POOL_SOURCE_OPERATOR_ORDER): + family = multispine_pool_module._SOURCE_OPERATOR_FAMILIES[operator_name] + output = f"fixture_source_output_{index:02d}" + output_families[family] = {"person": frozenset({output})} + + def apply( + available: Frame, + *, + name: str = operator_name, + column: str = output, + value: float = float(index + 1), + ) -> Frame: + calls.append(name) + assert "us_spine_assembly_manifest" not in available.metadata + assert not available.mass_log + person = available.table("person") + assert "person_support_channel" in person.columns + assert set(person["person_support_clone_index"]) == {0, 1} + assert person["PERIDNUM"].notna().all() + updated = person.copy() + updated[column] = value + return _replace_person(available, updated) + + operators[operator_name] = apply + + original_getitem = pd.DataFrame.__getitem__ + + def reject_source_channel_read( + table: pd.DataFrame, + key: object, + ) -> object: + keys = [key] if isinstance(key, str) else list(key) if isinstance( + key, (list, tuple) + ) else [] + if any(str(column).endswith("_support_channel") for column in keys): + raise AssertionError("population source channel was read") + return original_getitem(table, key) + + monkeypatch.setattr(pd.DataFrame, "__getitem__", reject_source_channel_read) + result = multispine_pool_module._run_source_operator_chain( + cloned, + operator_names=POOL_SOURCE_OPERATOR_ORDER, + operators=operators, + output_families=output_families, + ) + + assert calls == list(POOL_SOURCE_OPERATOR_ORDER) + assert result.receipt["operator_order"] == list(POOL_SOURCE_OPERATOR_ORDER) + assert result.receipt["cps_source_evidence"] == { + "column": "PERIDNUM", + "person_rows": 4, + } + for index, receipt in enumerate(result.receipt["suboperators"]): + assert receipt["order_index"] == index + assert receipt["operator"] == POOL_SOURCE_OPERATOR_ORDER[index] + assert receipt["pool_input_rows"]["person"] == 8 + assert receipt["cps_available_rows"]["person"] == 4 + assert receipt["operator_output_rows"]["person"] == 4 + assert receipt["merged_rows"]["person"] == 4 + assert receipt["operator_projection"] == { + "selection": "PERIDNUM", + "lineage_state_persisted": False, + } + + person = result.frame.table("person") + cps = person["PERIDNUM"].notna() + assert person.loc[cps, first_output].tolist() == [1.0] * 4 + assert sorted(person.loc[~cps, first_output].tolist()) == [ + 900.0, + 900.0, + 901.0, + 901.0, + ] + unavailable = "fixture_source_output_01" + assert person.loc[cps, unavailable].tolist() == [2.0] * 4 + assert person.loc[~cps, unavailable].isna().all() + assert result.frame.metadata == cloned.metadata + assert result.frame.mass_log == cloned.mass_log + + +def test_predictor_prep_fills_cps_rows_without_overwriting_acs_native() -> None: + asec = _source_frame() + asec_tables = {entity: asec.table(entity).copy() for entity in asec.entities} + asec_person = asec_tables["person"].drop(columns=["age"]) + asec_person["PERIDNUM"] = ["asec-1", "asec-2"] + asec_person["A_AGE"] = [31, 52] + asec_person["A_SEX"] = [1, 2] + asec_person["OI_VAL"] = [0.0, 25.0] + asec_person["OI_OFF"] = [0, 20] + asec_tables["person"] = asec_person + asec = Frame( + asec_tables, + asec.schema, + {"household": asec.weights_for("household")}, + asec.strata, + ) + + acs = _source_frame() + acs_tables = {entity: acs.table(entity).copy() for entity in acs.entities} + acs_tables["person"]["age"] = [70.0, 80.0] + acs_tables["person"]["is_female"] = [True, False] + acs = Frame( + acs_tables, + acs.schema, + {"household": acs.weights_for("household")}, + acs.strata, + ) + cloned = clone_us_frame_for_puf_support( + assemble_spines( + {"asec": asec, "acs": acs}, + household_mass_shares={"asec": 0.5, "acs": 0.5}, + ) + ) + + result = prepare_multispine_puf_predictors(cloned) + + person = result.frame.table("person") + cps = person["PERIDNUM"].notna() + assert sorted(person.loc[cps, "age"].tolist()) == [31.0, 31.0, 52.0, 52.0] + assert sorted(person.loc[~cps, "age"].tolist()) == [70.0, 70.0, 80.0, 80.0] + assert sorted(person.loc[cps, "is_female"].tolist()) == [ + False, + False, + True, + True, + ] + assert sorted(person.loc[~cps, "is_female"].tolist()) == [ + False, + False, + True, + True, + ] + assert result.receipt["operator_order"] == ["derive_us_cps_carried_inputs"] def test_schedule_d_derivation_preserves_existing_values_and_receipt() -> None: @@ -329,6 +670,7 @@ def _assembled_cloned_with_partial_take_up() -> Frame: acs = _source_frame() tables = {entity: acs.table(entity).copy() for entity in acs.entities} tables["spm_unit"]["takes_up_housing_assistance_if_eligible"] = [False, True] + tables["person"]["takes_up_medicare_if_eligible"] = [True, False] acs = Frame( tables, acs.schema, @@ -374,9 +716,9 @@ def test_pool_seed_stage_preserves_inputs_and_receipts_disclosed_defaults() -> N assert tanf["provenance_kind"] == "administrative_seed_or_preserved_input" medicare = result.receipt["programs"]["takes_up_medicare_if_eligible"] assert medicare["provenance_kind"] == ( - "preserved_input_or_disclosed_engine_default" + "transferred_or_preserved_input" ) - assert medicare["defaulted_rows"] == 4 + assert medicare["defaulted_rows"] == 0 spm = result.frame.table("spm_unit") source_id = support_source_id_column("spm_unit") diff --git a/packages/populace-build/tests/test_us_multispine_pool_tool.py b/packages/populace-build/tests/test_us_multispine_pool_tool.py index e8b672ab..6f4a4275 100644 --- a/packages/populace-build/tests/test_us_multispine_pool_tool.py +++ b/packages/populace-build/tests/test_us_multispine_pool_tool.py @@ -2,11 +2,14 @@ from __future__ import annotations +import ast import hashlib import importlib.util +import inspect import json import sys from collections.abc import Callable +from dataclasses import replace from pathlib import Path from types import ModuleType @@ -15,9 +18,12 @@ import pytest import populace.build.us_runtime.acs_transfer as acs_transfer_module -from populace.build.gates import GateReport +from populace.build.gates import GateReport, GateResult from populace.build.us_runtime.acs_transfer import transfer_acs_inputs from populace.build.us_runtime.multispine_pool import PoolStageOutput +from populace.build.us_runtime.operator_boundary import ( + PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES, +) from populace.build.us_runtime.puf_support import ( PUF_SUPPORT_MAX_CLONE_SAFE_SOURCE_ID, ) @@ -53,7 +59,10 @@ def _source_frame(*, measured_offset: float = 0.0) -> Frame: "person_spm_unit_id": ids, "person_family_id": ids, "person_marital_unit_id": ids, - "age": np.asarray([30.0, 50.0]), + "A_AGE": np.asarray([30.0, 50.0]), + "A_SEX": np.asarray([1, 2], dtype=np.int64), + "PERIDNUM": np.asarray(["1", "2"], dtype=object), + "source_year": np.asarray([2024, 2024], dtype=np.int64), "measured": np.asarray([1.0, 2.0]) + measured_offset, } ) @@ -136,7 +145,6 @@ def predict(self, frame: pd.DataFrame) -> pd.DataFrame: def _transfer_source_frame(targets: list[float]) -> Frame: frame = _source_frame() tables = {entity: frame.table(entity).copy() for entity in frame.entities} - tables["person"]["is_female"] = np.asarray([False, True], dtype=bool) tables["person"]["fixture_transfer"] = np.asarray(targets, dtype=np.float64) tables["household"]["state_fips"] = np.asarray([6, 36], dtype=np.int64) return Frame( @@ -204,6 +212,7 @@ def simulate(person: pd.DataFrame) -> None: assert result.agreement_gate.details["tolerances"] == { "incidence_ratio_bounds": [0.8, 1.25], "max_quantile_envelope_distance": 0.25, + "max_categorical_total_variation_distance": 0.25, } assert "ssi" not in result.frame.table("person") return result @@ -219,9 +228,10 @@ def _output_context( verified_inputs = {} for index, role in enumerate( ( - "asec_pre_clone", + "asec_raw_stage", "acs_household", "acs_person", + "acs_rent_donor", "processed_puf", "puf_source_year", ), @@ -240,8 +250,9 @@ def _output_context( loaded = pool_tool._LoadedInputs( asec=_source_frame(), acs=_source_frame(measured_offset=99.0), + acs_rent_donor=pd.DataFrame({"fixture": [1]}), puf_donor=pd.DataFrame({"RECID": [1]}), - asec_checkpoint={"artifact": "fixture-pre-clone"}, + asec_raw_stage_checkpoint={"artifact": "fixture-raw-stage"}, acs_build={"artifact": "fixture-unit-frame"}, acs_native_inputs={"person": {"age": {"source": "fixture"}}}, puf_donor_build={"artifact": "fixture-donor"}, @@ -296,9 +307,13 @@ def _assert_publication_tombstone( "simulation_ready": False, "status": "publication_in_progress", } + with pytest.raises(ValueError, match="not simulation-ready"): + pool_tool.load_simulation_ready_us_multispine_pool_manifest( + outputs.manifest + ) -def test_parser_exposes_only_five_pinned_inputs_and_out( +def test_parser_exposes_only_six_pinned_inputs_and_out( pool_tool: ModuleType, ) -> None: parser = pool_tool._parser() @@ -306,9 +321,10 @@ def test_parser_exposes_only_five_pinned_inputs_and_out( action.dest: action for action in parser._actions if action.dest != "help" } pairs = ( - ("asec_pre_clone_h5", "asec_pre_clone_h5_sha256"), + ("asec_raw_stage_h5", "asec_raw_stage_h5_sha256"), ("acs_household_zip", "acs_household_zip_sha256"), ("acs_person_zip", "acs_person_zip_sha256"), + ("acs_rent_h5", "acs_rent_h5_sha256"), ("puf_h5", "puf_h5_sha256"), ("puf_source_year_csv", "puf_source_year_csv_sha256"), ) @@ -335,15 +351,111 @@ def test_parser_exposes_only_five_pinned_inputs_and_out( ) +def test_pool_tool_structurally_accepts_only_the_raw_stage_loader( + pool_tool: ModuleType, +) -> None: + source = inspect.getsource(pool_tool) + tree = ast.parse(source) + imported = { + alias.name + for node in ast.walk(tree) + if isinstance(node, ast.ImportFrom) + for alias in node.names + } + called = { + node.func.id + for node in ast.walk(tree) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + } + + assert "load_asec_raw_stage_checkpoint" in imported + assert "load_asec_raw_stage_checkpoint" in called + assert "load_asec_pre_clone_checkpoint" not in imported + assert "load_asec_pre_clone_checkpoint" not in called + assert "pre_clone_enrichment" not in source + + +def test_pool_imputation_wires_full_source_chain_after_primary_and_tail( + pool_tool: ModuleType, +) -> None: + source = inspect.getsource(pool_tool._impute_pool) + tree = ast.parse(source) + expected = ( + "prepare_multispine_puf_predictors", + "_initialize_or_resume_primary_qrf", + "run_primary_puf_qrf_chain", + "finalize_primary_puf_qrf_chain", + "transfer_puf_capital_gains_tail", + "complete_multispine_source_inputs", + "pool_transfer_target_families", + "transfer_acs_inputs", + ) + calls = sorted( + ( + node.lineno, + node.func.id, + ) + for node in ast.walk(tree) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id in expected + ) + + assert tuple(name for _line, name in calls) == expected + + +def test_direct_pool_fixtures_are_operator_free_before_assembly() -> None: + frame = _source_frame() + for entities in PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES.values(): + for entity, columns in entities.items(): + assert set(frame.table(entity)).isdisjoint(columns) + + +@pytest.mark.parametrize( + ("family", "entity", "column"), + [ + (family, entity, sorted(columns)[0]) + for family, entities in PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES.items() + for entity, columns in entities.items() + ], +) +def test_each_operator_output_family_is_rejected_before_assembly( + pool_tool: ModuleType, + family: str, + entity: str, + column: str, +) -> None: + frame = _source_frame() + tables = {name: frame.table(name).copy() for name in frame.entities} + tables[entity][column] = 0.0 + contaminated = Frame( + tables, + frame.schema, + {"household": frame.weights_for("household")}, + frame.strata, + ) + + with pytest.raises( + ValueError, + match=rf"fixture {family}.*{column}|fixture.*{column}", + ): + pool_tool.assert_operator_free_source_frame( + contaminated, + label=f"fixture {family}", + ) + + def test_sha_mismatch_refuses_before_loading_or_writing( pool_tool: ModuleType, monkeypatch: pytest.MonkeyPatch, tmp_path: Path, ) -> None: source_paths = { - "asec-pre-clone-h5": tmp_path / "asec.h5", + "asec-raw-stage-h5": tmp_path / "asec.h5", "acs-household-zip": tmp_path / "household.zip", "acs-person-zip": tmp_path / "person.zip", + "acs-rent-h5": tmp_path / "rent.h5", "puf-h5": tmp_path / "puf.h5", "puf-source-year-csv": tmp_path / "puf.csv", } @@ -366,10 +478,15 @@ def unexpected_write(*_args, **_kwargs): argv: list[str] = [] for option, path in source_paths.items(): argv.extend([f"--{option}", str(path)]) - argv.extend([f"--{option}-sha256", "0" * 64]) + digest = ( + pool_tool.ACS_2022_RENT_ARTIFACT_SHA256 + if option == "acs-rent-h5" + else "0" * 64 + ) + argv.extend([f"--{option}-sha256", digest]) argv.extend(["--out", str(output)]) - with pytest.raises(ValueError, match="ASEC pre-clone.*SHA-256 mismatch"): + with pytest.raises(ValueError, match="ASEC raw-stage.*SHA-256 mismatch"): pool_tool.main(argv) assert called == {"load": False, "write": False} @@ -450,9 +567,15 @@ def test_wired_path_uses_real_raw_preserving_transfer_before_gate( transfer_receipts = [] def impute(frame: Frame) -> PoolStageOutput: + person = frame.table("person").copy() + person["age"] = pd.to_numeric(person["A_AGE"], errors="raise") + person["is_female"] = ( + pd.to_numeric(person["A_SEX"], errors="raise") == 2 + ) + prepared = _replace_person(frame, person) transferred = transfer_acs_inputs( - frame, - frame, + prepared, + prepared, target_families={"person": {"fixture": ("fixture_transfer",)}}, seed=0, n_estimators=3, @@ -554,6 +677,72 @@ def test_red_outputs_preserve_receipts_and_exclude_simulation_output( assert metadata["publication_run_id"] == manifest["publication_run_id"] +def test_ready_reader_binds_manifest_h5_and_diagnostics_to_one_run( + pool_tool: ModuleType, + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + result, outputs, verified_inputs, source_manifest, loaded = _output_context( + pool_tool, + tmp_path, + ) + ready = replace( + result, + agreement_gate=GateResult("us_spine_agreement", True), + ) + pool_tool._write_outputs( + ready, + outputs=outputs, + verified_inputs=verified_inputs, + acs_source_manifest=source_manifest, + loaded=loaded, + ) + + manifest = pool_tool.load_simulation_ready_us_multispine_pool_manifest( + outputs.manifest + ) + + assert manifest["simulation_ready"] is True + assert ( + manifest["agreement_diagnostics"]["publication_run_id"] + == manifest["publication_run_id"] + ) + + +def test_ready_reader_rejects_manifest_h5_run_id_mismatch( + pool_tool: ModuleType, + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + result, outputs, verified_inputs, source_manifest, loaded = _output_context( + pool_tool, + tmp_path, + ) + ready = replace( + result, + agreement_gate=GateResult("us_spine_agreement", True), + ) + pool_tool._write_outputs( + ready, + outputs=outputs, + verified_inputs=verified_inputs, + acs_source_manifest=source_manifest, + loaded=loaded, + ) + manifest = json.loads(outputs.manifest.read_text(encoding="utf-8")) + manifest["publication_run_id"] = "substituted-run" + manifest["pool_h5"]["publication_run_id"] = "substituted-run" + outputs.manifest.write_text( + json.dumps(manifest), + encoding="utf-8", + ) + + with pytest.raises(ValueError, match="H5.*run ID does not match"): + pool_tool.load_simulation_ready_us_multispine_pool_manifest( + outputs.manifest + ) + + def test_h5_publication_failure_invalidates_stale_green_manifest( pool_tool: ModuleType, monkeypatch: pytest.MonkeyPatch, diff --git a/packages/populace-build/tests/test_us_puf_support_base_builder.py b/packages/populace-build/tests/test_us_puf_support_base_builder.py index 70ed3d37..a7d7487a 100644 --- a/packages/populace-build/tests/test_us_puf_support_base_builder.py +++ b/packages/populace-build/tests/test_us_puf_support_base_builder.py @@ -9,6 +9,7 @@ import pytest from populace.build import FitWeightRecord +from populace.build.outer_stage_runtime import frame_identity from populace.build.us_runtime import ( US_PUF_SUPPORT_FIT_NAME, clone_us_frame_for_puf_support, @@ -79,6 +80,150 @@ def _minimal_us_frame() -> Frame: return Frame(tables, US_SCHEMA, weights, strata) +def _raw_asec_frame() -> Frame: + source = _minimal_us_frame() + tables = {entity: source.table(entity).copy() for entity in source.entities} + person = tables["person"].drop(columns=["employment_income_before_lsr"]) + person["source_year"] = np.asarray([2022, 2022, 2022], dtype=np.int64) + person["source_household_id"] = np.asarray([101, 101, 202], dtype=np.int64) + person["source_person_id"] = np.asarray( + [f"{value:022d}" for value in (1, 2, 3)], + dtype=object, + ) + person["source_row_id"] = np.asarray([0, 1, 2], dtype=np.int64) + person["PERIDNUM"] = person["source_person_id"].to_numpy() + person["P_SEQ"] = np.asarray([1, 2, 1], dtype=np.int64) + person["A_LINENO"] = np.asarray([1, 2, 1], dtype=np.int64) + person["A_AGE"] = np.asarray([31, 29, 50], dtype=np.int64) + tables["person"] = person + return Frame( + tables, + source.schema, + { + entity: source.weights_for(entity) + for entity in source.weighted_entities + }, + pd.Series(["asec_2022"] * 3, name="stratum"), + ) + + +def _weeks_source() -> pd.DataFrame: + source = pd.DataFrame( + { + "PH_SEQ": [101, 101, 202], + "P_SEQ": [1, 2, 1], + "A_LINENO": [1, 2, 1], + "PERIDNUM": [f"{value:022d}" for value in (1, 2, 3)], + "LKWEEKS": [7, -1, 12], + } + ) + source.attrs["source_audit"] = {"rows": 3} + return source + + +def _education_source() -> pd.DataFrame: + source = pd.DataFrame( + { + "source_year": [2022, 2022, 2022], + "PH_SEQ": [101, 101, 202], + "P_SEQ": [1, 2, 1], + "A_LINENO": [1, 2, 1], + "PERIDNUM": [f"{value:022d}" for value in (1, 2, 3)], + "ED_VAL": [0.0, 500.0, 1_000.0], + } + ) + source.attrs["source_audit"] = {2022: {"rows": 3}} + return source + + +def _pooled_source_receipt(tmp_path: Path) -> dict[str, object]: + return { + "kind": "pooled_asec", + "target_year": 2022, + "sources": [ + { + "year": 2022, + "path": str((tmp_path / "asec_2022.h5").resolve()), + "sha256": "a" * 64, + "share": 1.0, + "max_households": None, + } + ], + "support_spine_spec": None, + "metadata": {"weighted_person_population": 400.0}, + } + + +def _with_person_column(frame: Frame, column: str, values: np.ndarray) -> Frame: + tables = {entity: frame.table(entity).copy() for entity in frame.entities} + tables["person"][column] = values + return Frame( + tables, + frame.schema, + { + entity: frame.weights_for(entity) + for entity in frame.weighted_entities + }, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + + +def _raw_stage_args(builder, tmp_path: Path): + return builder._parse_args( + [ + "--asec-h5", + f"2022={tmp_path / 'asec_2022.h5'}", + "--target-year", + "2022", + "--puf-h5", + str(tmp_path / "puf.h5"), + "--asec-2023-weeks-unemployed-source", + str(tmp_path / "asec_weeks.zip"), + "--asec-education-source", + f"2022={tmp_path / 'asec_education.zip'}", + "--out", + str(tmp_path / "out"), + "--without-block-ladder", + "--stage", + "source_construction", + "--checkpoint-dir", + str(tmp_path / "checkpoints"), + ] + ) + + +def _patch_raw_stage_sources( + monkeypatch: pytest.MonkeyPatch, + builder, + *, + frame: Frame, + source_receipt: dict[str, object], +) -> None: + monkeypatch.setattr( + builder, + "_load_base_frame_from_args", + lambda _args: (frame, source_receipt), + ) + monkeypatch.setattr( + builder, + "load_asec_2023_weeks_unemployed_source", + lambda _path: _weeks_source(), + ) + monkeypatch.setattr( + builder, + "load_asec_education_assistance_sources", + lambda _paths, *, income_years: _education_source(), + ) + monkeypatch.setattr( + builder, + "_builder_code_identity", + lambda: {"source_sha256": "raw-stage-fixture"}, + ) + monkeypatch.setattr(builder, "_sha256", lambda _path: "f" * 64) + + def _support_donor() -> pd.DataFrame: return pd.DataFrame( { @@ -532,6 +677,178 @@ def test_reconciled_outer_pipeline_order_is_locked() -> None: assert tuple(name for name, _boundaries in builder.STAGE_BOUNDARIES) == expected +def test_raw_stage_copy_adds_only_exact_source_mappings( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + builder = _load_support_builder_module() + args = _raw_stage_args(builder, tmp_path) + source = _raw_asec_frame() + before = frame_identity(source) + monkeypatch.setattr( + builder, + "load_asec_2023_weeks_unemployed_source", + lambda _path: _weeks_source(), + ) + monkeypatch.setattr( + builder, + "load_asec_education_assistance_sources", + lambda _paths, *, income_years: _education_source(), + ) + + raw, mappings = builder._asec_raw_source_mapping_frame( + args, + source, + weeks_path=tmp_path / "asec_weeks.zip", + ) + + assert frame_identity(source) == before + assert "LKWEEKS" not in source.table("person") + assert "ED_VAL" not in source.table("person") + assert raw.table("person")["LKWEEKS"].tolist() == [7.0, -1.0, 12.0] + assert raw.table("person")["ED_VAL"].tolist() == [0.0, 500.0, 1_000.0] + assert set(mappings) == {"ED_VAL", "LKWEEKS"} + assert all( + mapping["operation"] == "exact_source_join" + and mapping["join_keys"] == ["source_year", "PERIDNUM"] + for mapping in mappings.values() + ) + builder.assert_operator_free_source_frame(raw, label="raw-stage fixture") + + +def test_pooled_source_stage_dual_exports_without_changing_legacy_checkpoints( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + builder = _load_support_builder_module() + args = _raw_stage_args(builder, tmp_path) + source = _raw_asec_frame() + _patch_raw_stage_sources( + monkeypatch, + builder, + frame=source, + source_receipt=_pooled_source_receipt(tmp_path), + ) + + def add_age(frame: Frame, **_kwargs) -> Frame: + return _with_person_column( + frame, + "age", + frame.table("person")["A_AGE"].to_numpy(), + ) + + baseline_runtime = builder.StageRuntime( + tmp_path / "legacy-baseline", + builder.OUTER_STAGE_PIPELINE, + run_config=builder._stage_run_config(args), + ) + baseline_source = baseline_runtime.complete("source_construction", source) + baseline_enriched = baseline_runtime.complete( + "pre_clone_enrichment", + add_age(source), + ) + + monkeypatch.setattr(builder, "derive_us_cps_carried_inputs", add_age) + identity_transforms = ( + "with_us_prior_year_income_inputs", + "with_us_relationship_inputs", + "with_us_medicare_take_up_input", + "with_us_eligibility_inputs", + "with_us_pregnancy_inputs", + "with_us_wic_claim_input", + "with_us_child_support_inputs", + "with_us_disability_benefits", + "with_us_workers_compensation", + "with_us_weeks_unemployed", + "with_us_childcare_inputs", + "with_us_energy_subsidy_input", + "with_us_retirement_contribution_inputs", + "with_us_retirement_distribution_inputs", + "with_us_immigration_inputs", + ) + for name in identity_transforms: + monkeypatch.setattr(builder, name, lambda value, **_kwargs: value) + passing_gate = SimpleNamespace(passed=True, failures=(), details={}) + for name in ( + "us_relationship_inputs_signal_gate", + "us_medicare_take_up_signal_gate", + "us_housing_inputs_signal_gate", + "us_eligibility_inputs_signal_gate", + "us_pregnancy_signal_gate", + "us_wic_claim_signal_gate", + ): + monkeypatch.setattr(builder, name, lambda _frame: passing_gate) + + builder._run_outer_stage(args) + runtime = builder.StageRuntime( + args.checkpoint_dir, + builder.OUTER_STAGE_PIPELINE, + run_config=builder._stage_run_config(args), + ) + source_checkpoint = runtime.load("source_construction") + assert frame_identity(source_checkpoint.frame) == frame_identity(source) + assert source_checkpoint.path.read_bytes() == baseline_source.path.read_bytes() + assert "ED_VAL" not in source_checkpoint.frame.table("person") + assert "LKWEEKS" not in source_checkpoint.frame.table("person") + + raw_path = args.checkpoint_dir / builder.ASEC_RAW_STAGE_CHECKPOINT_FILENAME + raw, raw_metadata = builder.load_asec_raw_stage_checkpoint(raw_path) + assert raw_metadata["stage"] == "raw_source_mapping" + assert raw.table("person")["LKWEEKS"].tolist() == [7.0, -1.0, 12.0] + assert raw.table("person")["ED_VAL"].tolist() == [0.0, 500.0, 1_000.0] + assert "age" not in raw.table("person") + assert [path.name for path in args.checkpoint_dir.glob("*.frame.h5")] == [ + "000_source_construction.frame.h5" + ] + + args.stage = "pre_clone_enrichment" + builder._run_outer_stage(args) + enriched_checkpoint = runtime.load("pre_clone_enrichment") + enriched = enriched_checkpoint.frame + assert enriched_checkpoint.path.read_bytes() == baseline_enriched.path.read_bytes() + assert enriched.table("person")["age"].tolist() == [31, 29, 50] + assert "ED_VAL" not in enriched.table("person") + assert "LKWEEKS" not in enriched.table("person") + assert sorted( + path.name for path in args.checkpoint_dir.glob("*.frame.h5") + ) == [ + "000_source_construction.frame.h5", + "001_pre_clone_enrichment.frame.h5", + ] + + +def test_completed_source_stage_repairs_raw_auxiliary_without_rewriting_legacy( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + builder = _load_support_builder_module() + args = _raw_stage_args(builder, tmp_path) + source = _raw_asec_frame() + _patch_raw_stage_sources( + monkeypatch, + builder, + frame=source, + source_receipt=_pooled_source_receipt(tmp_path), + ) + + builder._run_outer_stage(args) + raw_path = args.checkpoint_dir / builder.ASEC_RAW_STAGE_CHECKPOINT_FILENAME + source_path = args.checkpoint_dir / "000_source_construction.frame.h5" + context_path = args.checkpoint_dir / "stage_run_context.json" + expected_raw = raw_path.read_bytes() + expected_source = source_path.read_bytes() + expected_context = context_path.read_bytes() + raw_path.unlink() + + builder._run_outer_stage(args) + + assert raw_path.read_bytes() == expected_raw + assert source_path.read_bytes() == expected_source + assert context_path.read_bytes() == expected_context + repaired, _metadata = builder.load_asec_raw_stage_checkpoint(raw_path) + assert repaired.table("person")["ED_VAL"].tolist() == [0.0, 500.0, 1_000.0] + + def test_source_and_preclone_stages_round_trip_design_weight_kind( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, @@ -615,6 +932,9 @@ def test_source_and_preclone_stages_round_trip_design_weight_kind( "pre_clone_enrichment", ) assert loaded.frame.weights_for("household").kind is WeightKind.DESIGN + assert not ( + args.checkpoint_dir / builder.ASEC_RAW_STAGE_CHECKPOINT_FILENAME + ).exists() def test_weeks_post_clone_rejects_source_content_drift( diff --git a/packages/populace-build/tests/test_us_spine_agreement.py b/packages/populace-build/tests/test_us_spine_agreement.py index a0dbc3c9..65b8590c 100644 --- a/packages/populace-build/tests/test_us_spine_agreement.py +++ b/packages/populace-build/tests/test_us_spine_agreement.py @@ -12,6 +12,7 @@ declared_acs_transfer_target_families, ) from populace.build.us_runtime.spine_agreement import ( + DEFAULT_CATEGORICAL_TOTAL_VARIATION_TOLERANCE, DEFAULT_INCIDENCE_RATIO_BOUNDS, DEFAULT_QUANTILE_ENVELOPE_TOLERANCE, DEFAULT_SPINE_AGREEMENT_QUANTILES, @@ -94,6 +95,7 @@ def test_default_registry_exactly_covers_chartered_agreement_surface() -> None: assert DEFAULT_INCIDENCE_RATIO_BOUNDS == (0.8, 1.25) assert DEFAULT_SPINE_AGREEMENT_QUANTILES == (0.10, 0.25, 0.50, 0.75, 0.90) assert DEFAULT_QUANTILE_ENVELOPE_TOLERANCE == 0.25 + assert DEFAULT_CATEGORICAL_TOTAL_VARIATION_TOLERANCE == 0.25 def test_registry_normalizes_and_merges_qrf_batches() -> None: @@ -207,6 +209,72 @@ def test_gate_batches_incidence_and_quantile_failures_across_columns() -> None: ) +def test_gate_checks_joint_categorical_distribution_not_only_marginals() -> None: + frame = _Frame( + { + "person": pd.DataFrame( + { + "person_support_channel": [ + "acs", + "acs", + "asec", + "asec", + ], + "ssn_card_type": [ + "CITIZEN", + "NONE", + "CITIZEN", + "NONE", + ], + "immigration_status_str": [ + "CITIZEN", + "UNDOCUMENTED", + "UNDOCUMENTED", + "CITIZEN", + ], + } + ) + }, + {"person": [1.0, 1.0, 1.0, 1.0]}, + ) + spec = SpineAgreementSpec( + entity="person", + family="source_operator_immigration", + columns=("ssn_card_type", "immigration_status_str"), + joint_categorical_groups=( + ("ssn_card_type", "immigration_status_str"), + ), + ) + + result = spine_agreement_gate(frame, registry=(spec,)) + + assert not result.passed + assert result.failures == ( + "person/source_operator_immigration/" + "joint[ssn_card_type,immigration_status_str]/acs_vs_asec: weighted " + "categorical total-variation distance 1 exceeds 0.25.", + ) + comparisons = result.details["comparisons"] + assert comparisons[ + "person/source_operator_immigration/ssn_card_type/acs_vs_asec" + ]["categorical_total_variation_distance"] == 0.0 + assert comparisons[ + "person/source_operator_immigration/immigration_status_str/acs_vs_asec" + ]["categorical_total_variation_distance"] == 0.0 + joint = comparisons[ + "person/source_operator_immigration/" + "joint[ssn_card_type,immigration_status_str]/acs_vs_asec" + ] + assert joint["categorical_total_variation_distance"] == 1.0 + assert result.details["checked_columns"] == 2 + assert result.details["checked_joint_categorical_groups"] == 1 + assert result.details["checked_spine_pairs"] == 3 + assert result.details["tested_spine_pairs"] == 3 + assert result.details["tolerances"][ + "max_categorical_total_variation_distance" + ] == 0.25 + + def test_default_registry_rejects_403_shaped_ssi_spine_disagreement() -> None: """The reviewer's default-registry repro must reach the SSI comparison.""" @@ -509,3 +577,10 @@ def test_malformed_registry_fails_closed() -> None: family="numeric", columns=["amount"], ) + with pytest.raises(ValueError, match="invalid joint categorical"): + SpineAgreementSpec( + entity="person", + family="categorical", + columns=("first", "second"), + joint_categorical_groups=(("first", "missing"),), + ) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 4c93a585..f30401dd 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -137,6 +137,7 @@ "medicaid_take_up.py", "misc_itemized.py", "nonzero_shares.py", + "operator_boundary.py", # Raw-stage validator; no population treatment. "org_wages.py", "parity_reference.py", "pregnancy.py", diff --git a/tools/build_us_multispine_pool.py b/tools/build_us_multispine_pool.py index b4078dd0..200c6aa7 100644 --- a/tools/build_us_multispine_pool.py +++ b/tools/build_us_multispine_pool.py @@ -47,9 +47,20 @@ transfer_acs_inputs, ) from populace.build.us_runtime.asec_checkpoint import ( - load_asec_pre_clone_checkpoint, + load_asec_raw_stage_checkpoint, +) +from populace.build.us_runtime.h5_io import ( + US_MULTISPINE_AGREEMENT_DIAGNOSTICS_ARTIFACT_KIND, + US_MULTISPINE_POOL_H5_ARTIFACT_KIND, + US_MULTISPINE_POOL_MANIFEST_ARTIFACT_KIND, + US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION, + load_simulation_ready_us_multispine_pool_manifest, + write_nullable_us_h5, +) +from populace.build.us_runtime.housing_inputs import ( + ACS_2022_RENT_ARTIFACT_SHA256, + load_acs_2022_rent_donor, ) -from populace.build.us_runtime.h5_io import write_nullable_us_h5 from populace.build.us_runtime.multispine_pool import ( POOL_HOUSEHOLD_MASS_SHARES, POOL_OPERATOR_ORDER, @@ -57,12 +68,17 @@ POOL_TIME_PERIOD, MultispinePoolResult, PoolStageOutput, + complete_multispine_source_inputs, derive_multispine_pool_inputs, materialize_multispine_agreement_outputs, pool_transfer_target_families, + prepare_multispine_puf_predictors, run_multispine_pool_path, seed_multispine_pool_inputs, ) +from populace.build.us_runtime.operator_boundary import ( + assert_operator_free_source_frame, +) from populace.build.us_runtime.puf_capital_gains_tail import ( transfer_puf_capital_gains_tail, validate_puf_capital_gains_tail_manifest, @@ -85,13 +101,14 @@ "POOL_MANIFEST_SCHEMA_VERSION", "PoolBuildOutputs", "build_multispine_pool", + "load_simulation_ready_us_multispine_pool_manifest", "main", ] -POOL_MANIFEST_SCHEMA_VERSION = 2 +POOL_MANIFEST_SCHEMA_VERSION = US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION """Schema version for the companion pool build manifest.""" -POOL_H5_ARTIFACT_KIND = "populace_us_multispine_input_pool" +POOL_H5_ARTIFACT_KIND = US_MULTISPINE_POOL_H5_ARTIFACT_KIND """Neutral H5 artifact kind; readiness is asserted only by the manifest.""" _PRIMARY_QRF_N_ESTIMATORS = 100 @@ -134,8 +151,9 @@ def to_manifest(self) -> dict[str, object]: class _LoadedInputs: asec: Frame acs: Frame + acs_rent_donor: pd.DataFrame puf_donor: pd.DataFrame - asec_checkpoint: Mapping[str, object] + asec_raw_stage_checkpoint: Mapping[str, object] acs_build: Mapping[str, object] acs_native_inputs: Mapping[str, Mapping[str, Any]] puf_donor_build: Mapping[str, object] @@ -152,16 +170,16 @@ def _sha256_argument(value: str) -> str: def _parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument( - "--asec-pre-clone-h5", + "--asec-raw-stage-h5", required=True, type=Path, - help="Input-complete ASEC pre_clone_enrichment Frame checkpoint.", + help="Operator-untouched ASEC raw_source_mapping Frame checkpoint.", ) parser.add_argument( - "--asec-pre-clone-h5-sha256", + "--asec-raw-stage-h5-sha256", required=True, type=_sha256_argument, - help="Expected SHA-256 of --asec-pre-clone-h5.", + help="Expected SHA-256 of --asec-raw-stage-h5.", ) parser.add_argument( "--acs-household-zip", @@ -187,6 +205,18 @@ def _parser() -> argparse.ArgumentParser: type=_sha256_argument, help="Expected SHA-256 of --acs-person-zip.", ) + parser.add_argument( + "--acs-rent-h5", + required=True, + type=Path, + help="Local canonical ACS 2022 rent-donor H5.", + ) + parser.add_argument( + "--acs-rent-h5-sha256", + required=True, + type=_sha256_argument, + help="Expected canonical SHA-256 of --acs-rent-h5.", + ) parser.add_argument( "--puf-h5", required=True, @@ -276,9 +306,10 @@ def _verify_inputs( outputs: PoolBuildOutputs, ) -> tuple[dict[str, _VerifiedInput], AcsSourceManifest]: source_paths = { - Path(args.asec_pre_clone_h5).resolve(), + Path(args.asec_raw_stage_h5).resolve(), Path(args.acs_household_zip).resolve(), Path(args.acs_person_zip).resolve(), + Path(args.acs_rent_h5).resolve(), Path(args.puf_h5).resolve(), Path(args.puf_source_year_csv).resolve(), } @@ -293,11 +324,17 @@ def _verify_inputs( raise ValueError(f"Pool outputs must not overwrite inputs: {collisions}.") acs_source_manifest = load_acs_source_manifest() + if args.acs_rent_h5_sha256 != ACS_2022_RENT_ARTIFACT_SHA256: + raise ValueError( + "ACS rent donor CLI pin differs from the canonical archived pin: " + f"got {args.acs_rent_h5_sha256}, expected " + f"{ACS_2022_RENT_ARTIFACT_SHA256}." + ) verified = { - "asec_pre_clone": _verify_file( - "ASEC pre-clone checkpoint", - args.asec_pre_clone_h5, - args.asec_pre_clone_h5_sha256, + "asec_raw_stage": _verify_file( + "ASEC raw-stage checkpoint", + args.asec_raw_stage_h5, + args.asec_raw_stage_h5_sha256, ), "acs_household": _verify_acs_file( "ACS household archive", @@ -311,6 +348,11 @@ def _verify_inputs( args.acs_person_zip_sha256, acs_source_manifest.artifact("person"), ), + "acs_rent_donor": _verify_file( + "ACS rent donor", + args.acs_rent_h5, + args.acs_rent_h5_sha256, + ), "processed_puf": _verify_file( "processed PUF H5", args.puf_h5, @@ -330,7 +372,9 @@ def _load_inputs( *, acs_source_manifest: AcsSourceManifest, ) -> _LoadedInputs: - asec, asec_checkpoint = load_asec_pre_clone_checkpoint(args.asec_pre_clone_h5) + asec, asec_raw_stage_checkpoint = load_asec_raw_stage_checkpoint( + args.asec_raw_stage_h5 + ) acs_source = AcsPumsSource( household_zip=args.acs_household_zip, person_zip=args.acs_person_zip, @@ -338,6 +382,7 @@ def _load_inputs( ) acs_frame, acs_build = build_acs_pums_unit_frame(acs_source) mapped_acs = map_acs_native_inputs(acs_frame) + acs_rent_donor = load_acs_2022_rent_donor(args.acs_rent_h5) donor_build: dict[str, object] = {} puf_donor = load_puf_tax_unit_donor( args.puf_h5, @@ -347,8 +392,9 @@ def _load_inputs( return _LoadedInputs( asec=asec, acs=mapped_acs.frame, + acs_rent_donor=acs_rent_donor, puf_donor=puf_donor, - asec_checkpoint=asec_checkpoint, + asec_raw_stage_checkpoint=asec_raw_stage_checkpoint, acs_build=acs_build, acs_native_inputs=mapped_acs.native_inputs, puf_donor_build=donor_build, @@ -430,11 +476,13 @@ def _impute_pool( frame: Frame, *, puf_donor: pd.DataFrame, + acs_rent_donor: pd.DataFrame, checkpoint_dir: Path, checkpoint_input_binding: Mapping[str, object], ) -> PoolStageOutput: + predictor_preparation = prepare_multispine_puf_predictors(frame) _initialize_or_resume_primary_qrf( - frame, + predictor_preparation.frame, puf_donor, checkpoint_dir, input_binding=checkpoint_input_binding, @@ -443,7 +491,7 @@ def _impute_pool( tail_bound_diagnostics: list[dict[str, object]] = [] with_primary_detail, primary_weight_kind = finalize_primary_puf_qrf_chain( - frame, + predictor_preparation.frame, checkpoint_dir, tail_bound_diagnostics=tail_bound_diagnostics, ) @@ -461,10 +509,14 @@ def _impute_pool( f"{tail_ceiling['positive_mass_five_x_ceiling']} <= " f"{tail_ceiling['positive_mass_five_x_target']}." ) + source_completion = complete_multispine_source_inputs( + with_tail, + acs_rent_donor=acs_rent_donor, + ) transfer_families = pool_transfer_target_families() transferred = transfer_acs_inputs( - with_tail, - with_tail, + source_completion.frame, + source_completion.frame, target_families=transfer_families, donor_channel=ACS_DONOR_CHANNEL_AUTO, seed=POOL_RANDOM_SEED, @@ -488,6 +540,10 @@ def _impute_pool( return PoolStageOutput( transferred.frame, { + "source_operator_chain": { + "predictor_preparation": dict(predictor_preparation.receipt), + "post_primary_completion": dict(source_completion.receipt), + }, "primary_puf_qrf": { "checkpoint_manifest": _read_json_object(qrf_manifest_path), "checkpoint_manifest_sha256": _file_sha256(qrf_manifest_path), @@ -516,8 +572,14 @@ def build_multispine_pool( acs: Frame, *, puf_donor: pd.DataFrame, + acs_rent_donor: pd.DataFrame | None = None, primary_qrf_checkpoint_dir: Path, checkpoint_input_binding: Mapping[str, object] | None = None, + source_native_inputs: Mapping[ + str, + Mapping[str, Mapping[str, Any]], + ] + | None = None, impute: PoolOperator | None = None, derive: PoolOperator = derive_multispine_pool_inputs, seed: PoolOperator = seed_multispine_pool_inputs, @@ -530,17 +592,33 @@ def build_multispine_pool( its registry or fixed tolerances. """ + native_inputs = source_native_inputs or {} + assert_operator_free_source_frame( + asec, + label="ASEC raw-stage pool input", + native_inputs=native_inputs.get("asec"), + ) + assert_operator_free_source_frame( + acs, + label="ACS native-mapped pool input", + native_inputs=native_inputs.get("acs"), + ) if impute is None: if checkpoint_input_binding is None: raise ValueError( "Production pool imputation requires a verified checkpoint " "input binding." ) + if acs_rent_donor is None: + raise ValueError( + "Production pool imputation requires the canonical ACS rent donor." + ) def impute_operator(frame: Frame) -> PoolStageOutput: return _impute_pool( frame, puf_donor=puf_donor, + acs_rent_donor=acs_rent_donor, checkpoint_dir=primary_qrf_checkpoint_dir, checkpoint_input_binding=checkpoint_input_binding, ) @@ -584,7 +662,7 @@ def _manifest_payload( "provenance_pins": { role: pin.to_manifest() for role, pin in verified_inputs.items() }, - "asec_pre_clone_checkpoint": loaded.asec_checkpoint, + "asec_raw_stage_checkpoint": loaded.asec_raw_stage_checkpoint, "acs_source_manifest": asdict(acs_source_manifest), "acs_pums_build": loaded.acs_build, "acs_native_inputs": loaded.acs_native_inputs, @@ -622,6 +700,7 @@ def _manifest_payload( "path": str(outputs.agreement_diagnostics.resolve()), "sha256": _file_sha256(outputs.agreement_diagnostics), "size_bytes": outputs.agreement_diagnostics.stat().st_size, + "publication_run_id": publication_run_id, }, "primary_qrf_checkpoint_dir": str(outputs.primary_qrf_checkpoint_dir.resolve()), "calibration": { @@ -639,7 +718,7 @@ def _publication_tombstone( publication_run_id: str, ) -> dict[str, object]: return { - "artifact_kind": "populace_us_multispine_pool_manifest", + "artifact_kind": US_MULTISPINE_POOL_MANIFEST_ARTIFACT_KIND, "schema_version": POOL_MANIFEST_SCHEMA_VERSION, "status": "publication_in_progress", "simulation_ready": False, @@ -693,7 +772,7 @@ def _write_outputs( publication_run_id=publication_run_id, ) diagnostics = { - "artifact_kind": "populace_us_multispine_agreement_diagnostics", + "artifact_kind": US_MULTISPINE_AGREEMENT_DIAGNOSTICS_ARTIFACT_KIND, "schema_version": POOL_MANIFEST_SCHEMA_VERSION, "simulation_ready": result.simulation_ready, "publication_run_id": publication_run_id, @@ -796,8 +875,10 @@ def main(argv: list[str] | None = None) -> int: loaded.asec, loaded.acs, puf_donor=loaded.puf_donor, + acs_rent_donor=loaded.acs_rent_donor, primary_qrf_checkpoint_dir=outputs.primary_qrf_checkpoint_dir, checkpoint_input_binding=_checkpoint_input_binding(verified_inputs), + source_native_inputs={"acs": loaded.acs_native_inputs}, ) _write_outputs( result, diff --git a/tools/build_us_puf_support_base.py b/tools/build_us_puf_support_base.py index 4fce9e31..fd126632 100644 --- a/tools/build_us_puf_support_base.py +++ b/tools/build_us_puf_support_base.py @@ -32,12 +32,23 @@ StageRuntime, assert_clone_expansion, assert_unchanged_identity, + frame_identity, ) from populace.build.source_manifest import SupportSpineSpec, load_support_spine_manifest from populace.build.source_runtime import SourceRuntimeConfig, run_source_stage from populace.build.stage_profile import profile_stage from populace.build.us_runtime import ( + ASEC_2023_WEEKS_UNEMPLOYED_MEMBER, + ASEC_2023_WEEKS_UNEMPLOYED_MEMBER_SHA256, ASEC_2023_WEEKS_UNEMPLOYED_SOURCE_SHA256, + ASEC_2023_WEEKS_UNEMPLOYED_SOURCE_YEAR, + ASEC_2023_WEEKS_UNEMPLOYED_ZIP_URL, + ASEC_EDUCATION_ASSISTANCE_ARCHIVES, + ASEC_RAW_STAGE_ARTIFACT_KIND, + ASEC_RAW_STAGE_CHECKPOINT_FILENAME, + ASEC_RAW_STAGE_OPERATOR_STATUS, + ASEC_RAW_STAGE_SCHEMA_VERSION, + ASEC_RAW_STAGE_STAGE, BASE_ASEC_SUPPORT_CHANNEL, CONGRESSIONAL_DISTRICT_VINTAGE_CROSSWALK_SHA256_ATTR, CONGRESSIONAL_DISTRICT_VINTAGE_TARGET_ATTR, @@ -52,6 +63,7 @@ US_SOURCE_MANIFEST, US_SUPPORT_SPINE_SPEC, AsecSource, + assert_operator_free_source_frame, build_pooled_asec_unit_frame, build_puf_e01000_reconciliation_basis, clone_us_frame_for_puf_support, @@ -59,12 +71,15 @@ congressional_district_distribution_from_ledger_facts, derive_us_cps_carried_inputs, fetch_asec_2023_weeks_unemployed_source, + fill_asec_2022_weeks_unemployed_source, + fill_asec_education_assistance_source, finalize_puf_e01000_reconciliation, impute_us_housing_assistance_to_puf_support, impute_us_puf_tax_detail_support, load_acs_2022_rent_donor, load_asec_2023_weeks_unemployed_source, load_asec_education_assistance_sources, + load_asec_raw_stage_checkpoint, load_congressional_district_vintage_crosswalk, load_us_block_ladder, puf_tax_unit_donor_from_arrays, @@ -1652,7 +1667,14 @@ def _run_outer_stage(args: argparse.Namespace) -> None: run_config=_stage_run_config(args), ) if args.stage in runtime.context.completed: - if args.stage == "final_export": + if args.stage == "source_construction" and args.asec_h5 is not None: + loaded = runtime.load("source_construction") + _ensure_asec_raw_stage_checkpoint( + args, + loaded.frame, + runtime.metadata["source_construction"], + ) + elif args.stage == "final_export": _repair_completed_final_stage(args, runtime) elif args.stage == PUF_CAPITAL_GAINS_TAIL_STAGE_NAME: _ensure_capital_gains_tail_manifest( @@ -1668,6 +1690,8 @@ def _run_outer_stage(args: argparse.Namespace) -> None: "source construction unexpectedly has a predecessor" ) frame, metadata = _source_construction_stage(args) + if args.asec_h5 is not None: + _ensure_asec_raw_stage_checkpoint(args, frame, metadata) runtime.complete(args.stage, frame, metadata=metadata) return @@ -1785,6 +1809,177 @@ def _source_construction_stage( } +def _ensure_asec_raw_stage_checkpoint( + args: argparse.Namespace, + source_frame: Frame, + source_metadata: Mapping[str, object], +) -> dict[str, object]: + """Verify or atomically repair the auxiliary operator-untouched artifact.""" + + if args.asec_h5 is None: + raise ValueError("ASEC raw-stage checkpoint requires pooled --asec-h5 inputs.") + source_receipt = source_metadata.get("base_source") + if not isinstance(source_receipt, Mapping) or ( + source_receipt.get("kind") != "pooled_asec" + ): + raise ValueError( + "ASEC raw-stage checkpoint requires the source-construction " + "pooled_asec receipt." + ) + raw_frame, raw_source_mappings = _asec_raw_source_mapping_frame( + args, + source_frame, + weeks_path=Path(str(source_metadata["weeks_unemployed_source_path"])), + ) + metadata: dict[str, object] = json.loads( + json.dumps( + { + "artifact_kind": ASEC_RAW_STAGE_ARTIFACT_KIND, + "identity": frame_identity(raw_frame).to_payload(), + "operator_status": ASEC_RAW_STAGE_OPERATOR_STATUS, + "pipeline_sha256": OUTER_STAGE_PIPELINE.sha256, + "raw_source_mappings": raw_source_mappings, + "schema_version": ASEC_RAW_STAGE_SCHEMA_VERSION, + "source_construction_identity": frame_identity( + source_frame + ).to_payload(), + "source_receipt": dict(source_receipt), + "stage": ASEC_RAW_STAGE_STAGE, + }, + allow_nan=False, + sort_keys=True, + ) + ) + checkpoint_path = args.checkpoint_dir / ASEC_RAW_STAGE_CHECKPOINT_FILENAME + if checkpoint_path.is_file(): + try: + existing_frame, existing_metadata = load_asec_raw_stage_checkpoint( + checkpoint_path + ) + except (OSError, TypeError, ValueError): + pass + else: + if existing_metadata == metadata and _frames_exactly_equal( + existing_frame, + raw_frame, + ): + return existing_metadata + + write_frame_checkpoint(checkpoint_path, raw_frame, metadata=metadata) + return metadata + + +def _frames_exactly_equal(left: Frame, right: Frame) -> bool: + """Compare every persisted Frame component without value coercion.""" + + if ( + left.schema != right.schema + or left.entities != right.entities + or left.links != right.links + or left.weighted_entities != right.weighted_entities + or left.mass_log != right.mass_log + or left.metadata != right.metadata + or not left.strata.equals(right.strata) + ): + return False + if any( + not left.table(entity).equals(right.table(entity)) + for entity in left.entities + ): + return False + if any(not left.link(name).equals(right.link(name)) for name in left.links): + return False + return all( + left.weights_for(entity).kind == right.weights_for(entity).kind + and np.array_equal( + left.weights_for(entity).values, + right.weights_for(entity).values, + ) + for entity in left.weighted_entities + ) + + +def _asec_raw_source_mapping_frame( + args: argparse.Namespace, + source_frame: Frame, + *, + weeks_path: Path, +) -> tuple[Frame, dict[str, object]]: + """Copy source construction and add only exact measured ASEC joins.""" + + weeks_source = load_asec_2023_weeks_unemployed_source(weeks_path) + education_source = load_asec_education_assistance_sources( + _asec_education_source_paths(args), + income_years=_pooled_income_years(args), + ) + tables = { + entity: source_frame.table(entity).copy(deep=True) + for entity in source_frame.entities + } + person = fill_asec_2022_weeks_unemployed_source( + tables["person"], + weeks_source, + ) + person = fill_asec_education_assistance_source(person, education_source) + tables["person"] = person + raw_frame = Frame( + tables, + source_frame.schema, + { + entity: Weights( + source_frame.weights_for(entity).values.copy(), + source_frame.weights_for(entity).kind, + ) + for entity in source_frame.weighted_entities + }, + source_frame.strata.copy(deep=True), + mass_log=source_frame.mass_log, + metadata=source_frame.metadata, + ) + assert_operator_free_source_frame( + raw_frame, + label="producer ASEC raw_source_mapping artifact", + ) + education_pins = [ + { + "income_year": income_year, + "locator": ASEC_EDUCATION_ASSISTANCE_ARCHIVES[income_year].zip_url, + "member": ASEC_EDUCATION_ASSISTANCE_ARCHIVES[income_year].member, + "member_sha256": ( + ASEC_EDUCATION_ASSISTANCE_ARCHIVES[income_year].member_sha256 + ), + "sha256": ASEC_EDUCATION_ASSISTANCE_ARCHIVES[income_year].zip_sha256, + } + for income_year in _pooled_income_years(args) + ] + return raw_frame, { + "ED_VAL": { + "audit": dict(education_source.attrs.get("source_audit", {})), + "column": "ED_VAL", + "entity": "person", + "join_keys": ["source_year", "PERIDNUM"], + "operation": "exact_source_join", + "source_pins": education_pins, + }, + "LKWEEKS": { + "audit": dict(weeks_source.attrs.get("source_audit", {})), + "column": "LKWEEKS", + "entity": "person", + "join_keys": ["source_year", "PERIDNUM"], + "operation": "exact_source_join", + "source_pins": [ + { + "income_year": ASEC_2023_WEEKS_UNEMPLOYED_SOURCE_YEAR, + "locator": ASEC_2023_WEEKS_UNEMPLOYED_ZIP_URL, + "member": ASEC_2023_WEEKS_UNEMPLOYED_MEMBER, + "member_sha256": ASEC_2023_WEEKS_UNEMPLOYED_MEMBER_SHA256, + "sha256": ASEC_2023_WEEKS_UNEMPLOYED_SOURCE_SHA256, + } + ], + }, + } + + def _pre_clone_enrichment_stage( args: argparse.Namespace, raw_base: Frame, From a9baed9f7686e9651a5463ad46516b1553450d91 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 22:53:37 -0400 Subject: [PATCH 30/94] Checkpoint final validation --- PROGRESS.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index cfa9284b..1f2820be 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -46,12 +46,16 @@ the deprecated-but-supported local ACS release producer. - Added a readiness loader that accepts only a green manifest whose publication run ID and digests match the H5 metadata and agreement diagnostics; the three interruption regressions now exercise that reader as well as the tombstone. +- Committed the reviewed assembly-first lineage and readiness integration in + `5b930ab`. +- Ran the complete `populace-build` suite: 3,232 passed, 85 skipped, with only + five pre-existing runtime/deprecation warnings. Repository-wide Ruff passes. - Established the required final constraint: this live checkpoint will be restored byte-for-byte to `origin/main` before handoff so no root journal ships in the PR diff. ## Next -- Commit the reviewed raw-boundary and readiness integration. -- Run the full `populace-build` suite and Ruff. -- Write the external handoff and restore `PROGRESS.md` to `origin/main`. +- Resolve any finding from the final independent adversarial audit. +- Write the external handoff. +- Restore `PROGRESS.md` to `origin/main` and commit the cleanup. From b421bfac334a6feb4eac2ff3cd35a479eb062b40 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 22:57:11 -0400 Subject: [PATCH 31/94] Bind raw artifact to source construction --- PROGRESS.md | 2 ++ .../populace/build/us_runtime/asec_checkpoint.py | 5 +++++ .../tests/test_us_asec_checkpoint.py | 15 +++++++++++++++ 3 files changed, 22 insertions(+) diff --git a/PROGRESS.md b/PROGRESS.md index 1f2820be..98078cbc 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -43,6 +43,8 @@ the deprecated-but-supported local ACS release producer. receipt, production wiring and the immutable 20-operator order are tested, and the pool-specific agreement registry covers every expanded transfer, take-up, SSI, and joint immigration surface. +- Bound the raw artifact's declared source-construction identity to the live + structural identity, with a substituted-identity regression. - Added a readiness loader that accepts only a green manifest whose publication run ID and digests match the H5 metadata and agreement diagnostics; the three interruption regressions now exercise that reader as well as the tombstone. diff --git a/packages/populace-build/src/populace/build/us_runtime/asec_checkpoint.py b/packages/populace-build/src/populace/build/us_runtime/asec_checkpoint.py index cc1e76d6..2e6b74b8 100644 --- a/packages/populace-build/src/populace/build/us_runtime/asec_checkpoint.py +++ b/packages/populace-build/src/populace/build/us_runtime/asec_checkpoint.py @@ -177,6 +177,11 @@ def load_asec_raw_stage_checkpoint( metadata["source_construction_identity"], label="ASEC raw-stage source-construction identity", ) + if source_construction_identity != actual_identity: + raise ValueError( + f"ASEC raw-stage checkpoint {checkpoint_path} no longer has the " + "source-construction structural identity it declares." + ) metadata["identity"] = stored_identity.to_payload() metadata["source_construction_identity"] = ( source_construction_identity.to_payload() diff --git a/packages/populace-build/tests/test_us_asec_checkpoint.py b/packages/populace-build/tests/test_us_asec_checkpoint.py index b76536d8..22037aaa 100644 --- a/packages/populace-build/tests/test_us_asec_checkpoint.py +++ b/packages/populace-build/tests/test_us_asec_checkpoint.py @@ -344,6 +344,21 @@ def test_raw_loader_rejects_identity_not_bound_to_frame(tmp_path: Path) -> None: load_asec_raw_stage_checkpoint(path) +def test_raw_loader_rejects_wrong_source_construction_identity( + tmp_path: Path, +) -> None: + path = tmp_path / "raw-wrong-source-identity.checkpoint.h5" + frame = _raw_us_frame() + metadata = _raw_binding(frame) + metadata["source_construction_identity"] = frame_identity( + _raw_us_frame(id_offset=100) + ).to_payload() + _write_checkpoint(path, frame, metadata=metadata) + + with pytest.raises(ValueError, match="source-construction structural identity"): + load_asec_raw_stage_checkpoint(path) + + _OPERATOR_OUTPUT_CASES = tuple( (family, entity, column) for family, by_entity in PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES.items() From 307174ab8d08e7613b7561473c80b3ced64b32b5 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 22:59:55 -0400 Subject: [PATCH 32/94] Checkpoint adversarial audit --- PROGRESS.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 98078cbc..973a3598 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -50,14 +50,15 @@ the deprecated-but-supported local ACS release producer. interruption regressions now exercise that reader as well as the tombstone. - Committed the reviewed assembly-first lineage and readiness integration in `5b930ab`. -- Ran the complete `populace-build` suite: 3,232 passed, 85 skipped, with only +- Ran the complete `populace-build` suite: 3,233 passed, 85 skipped, with only five pre-existing runtime/deprecation warnings. Repository-wide Ruff passes. +- Independent final adversarial review of committed `b421bfa` found no + remaining blocker across the four round-1 HOLD items. - Established the required final constraint: this live checkpoint will be restored byte-for-byte to `origin/main` before handoff so no root journal ships in the PR diff. ## Next -- Resolve any finding from the final independent adversarial audit. - Write the external handoff. - Restore `PROGRESS.md` to `origin/main` and commit the cleanup. From c94fe06300ac01d43af00086a26a2ca9c28e2f3c Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 23:00:29 -0400 Subject: [PATCH 33/94] Keep PROGRESS.md at origin/main --- PROGRESS.md | 92 ++++++++++++++++++++++------------------------------- 1 file changed, 38 insertions(+), 54 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 973a3598..f3158fdd 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,63 +2,47 @@ ## State -PR #583 round-1 HOLD remediation is in progress on -`multispine-pool-build-578`. The four blocking seams are the raw-stage producer -boundary, crash-safe pool publication, pandas string-read guard coverage, and -the deprecated-but-supported local ACS release producer. +Populace #516 whole-row donor outlier screen is complete on +`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515 +interim carve merged as #525). The `puf_tax_detail` donor now drops tax units +whose grouped raw mortgage interest reaches $10M before the #515 carve +(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T +of phantom mortgage-interest mass), with the checkpoint schema bumped to v3 +so post-carve pre-screen checkpoints rebuild. ## Done -- Confirmed the worktree starts clean at review head `a20e847`. -- Read `CLAUDE.md` and the full adversarial review at - `_buildo-runtime/reviews/sol_583.log`. -- Selected the GitNexus debugging and impact-analysis workflows for lineage - tracing and blast-radius checks. -- Attempted to build the GitNexus index; the managed lane denied its global - registry write, so direct source/call-site/test tracing is the active - fallback. Removed the generated untracked local cache. -- Extended the spine-blindness visitor over pandas string APIs and committed - the exact reviewer `.get()` and `.query()` mutations plus `.eval()`, - `.filter(items=...)`, and `.loc[...]` regressions in `884c255`. The focused - guard suite passes (11 tests) and Ruff is clean. -- Reworked pool publication to invalidate first, stage H5 and diagnostics - under one run ID, publish them atomically, and write the readiness manifest - last in `b75f352`. The three reviewer interruption points now retain a - non-ready tombstone; focused H5/pool coverage passes (17 tests). -- Restored the deprecated ACS staging CLI by delegating to the preserved - pre-shim implementation in `tools/_legacy` in `f8bcb3b`. Its local-release - recipe, summary/reviewed-null contract, helper compatibility, and legacy - suite pass (37 tests); focused Ruff is clean. -- Split the pooled ASEC producer at a dedicated, operator-untouched raw-stage - artifact; its exact `LKWEEKS`/`ED_VAL` mappings, artifact binding, resume - repair, complete operator-family exclusion, and unchanged legacy checkpoint - sequence are covered by regressions. -- Rewired the multispine pool to consume only that raw artifact, assemble and - clone first, run the full 20-operator source-input chain on raw-evidenced - rows, preserve ACS-native cells, and transfer the remaining nullable peer - inputs with explicit ownership. Consolidated blocker coverage passes, - including the legacy local-release path and structural guard. -- Closed the independent lineage-review findings: ACS native exceptions bind - exact mapping contracts, source-kernel projections carry no false full-pool - receipt, production wiring and the immutable 20-operator order are tested, - and the pool-specific agreement registry covers every expanded transfer, - take-up, SSI, and joint immigration surface. -- Bound the raw artifact's declared source-construction identity to the live - structural identity, with a substituted-identity regression. -- Added a readiness loader that accepts only a green manifest whose publication - run ID and digests match the H5 metadata and agreement diagnostics; the three - interruption regressions now exercise that reader as well as the tombstone. -- Committed the reviewed assembly-first lineage and readiness integration in - `5b930ab`. -- Ran the complete `populace-build` suite: 3,233 passed, 85 skipped, with only - five pre-existing runtime/deprecation warnings. Repository-wide Ruff passes. -- Independent final adversarial review of committed `b421bfa` found no - remaining blocker across the four round-1 HOLD items. -- Established the required final constraint: this live checkpoint will be - restored byte-for-byte to `origin/main` before handoff so no root journal - ships in the PR diff. +- Confirmed a clean starting worktree at `aef1c56`. +- Read the repository guidance and established the #515 donor carve as the + screen's required downstream boundary. +- Started source-level audits of every donor-frame consumer, checkpoint + validation, row-count pins, and existing donor-fact summaries. +- Attempted the requested GitNexus impact workflow; the managed filesystem + denied its global registry write. Its local index also exposed a broad + `build/` ignore mismatch, so the completed impact audit uses direct source + call sites and tests. +- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the + structural rationale and pinned-artifact receipts. +- Added a whole-row screen on grouped raw person `home_mortgage_interest` + after tax-unit assembly, before the #515 carve, with retained-index reset. +- Confirmed no downstream consumer pairs donor rows to the original HDF arrays + or carries a stale donor-length vector; values and weights always originate + from the same screened frame. +- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale + checkpoint regression track the live constant while retaining literal-v1 + corruptions. +- Added regression coverage for the exact grouped boundary, whole-row removal, + retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant. +- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets + 139; populace-data 138 with 1 skip. The directly affected tail-bound suite + adds 12 passes. Ruff format/check and `git diff --check` are clean. +- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line + audit, expected 208,611-row real-artifact effect, verification results, count + sweep, and deliberately untouched surfaces. ## Next -- Write the external handoff. -- Restore `PROGRESS.md` to `origin/main` and commit the cleanup. +- PR #527 review cycle, then merge. After both #525 and #527: rebuild the + base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a + run that holds per `us_critical_targets.py`. +- Root record-level ETL carve stays open on populace#515. From 31356f2ab2e5452b663ae89aa57513f1fb30e109 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 23:28:10 -0400 Subject: [PATCH 34/94] Round 2: the guard resolves static indirection and fails closed on opacity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sol round-2 bypasses: expression strings and column lists bound to names one assignment away survived the .query/.eval/.filter checks, as did f-strings interpolating a bound column name. The visitor now carries a constant-propagation scope alongside its bindings (string and string-list literals, function- and lambda-scoped), resolves those forms, and — the completeness rule — records any pandas expression it still cannot resolve as a fail-closed violation rather than passing it. The 54-module operator graph scans clean under the stricter rule; the reviewer's four bypasses and an opaque-argument case are committed self-tests. Co-Authored-By: Claude Fable 5 --- .../tests/test_us_spine_blindness.py | 168 ++++++++++++++++-- 1 file changed, 153 insertions(+), 15 deletions(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index f30401dd..b0d411cd 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -323,6 +323,63 @@ def _subscript_source_expression( ) +def _static_string_shape( + node: ast.AST, constants: list[dict[str, object]] +) -> str | None: + """_string_shape with one level of static constant propagation. + + Names bound to resolvable string literals in an enclosing scope resolve + to their value (including inside f-string interpolations), so + ``expr = '...'; df.query(expr)`` and ``df.query(f"{col} == 'x'")`` are + seen through. Anything still opaque returns None — and the caller + treats an opaque pandas expression as a violation (fail-closed). + """ + + if isinstance(node, ast.Name): + for scope in reversed(constants): + if node.id in scope: + value = scope[node.id] + return value if isinstance(value, str) else None + return None + if isinstance(node, ast.FormattedValue): + inner = _static_string_shape(node.value, constants) + return inner if inner is not None else "*" + if isinstance(node, ast.JoinedStr): + pieces = [_static_string_shape(value, constants) for value in node.values] + if any(piece is None for piece in pieces): + return None + return "".join(piece for piece in pieces if piece is not None) + if isinstance(node, ast.BinOp) and isinstance(node.op, ast.Add): + left = _static_string_shape(node.left, constants) + right = _static_string_shape(node.right, constants) + if left is not None and right is not None: + return left + right + return None + return _string_shape(node) + + +def _static_string_list( + node: ast.AST, constants: list[dict[str, object]] +) -> list[str] | None: + """Resolve a static list/tuple of strings, through one Name binding.""" + + if isinstance(node, ast.Name): + for scope in reversed(constants): + if node.id in scope: + value = scope[node.id] + return value if isinstance(value, list) else None + return None + if isinstance(node, (ast.List, ast.Tuple)): + items: list[str] = [] + for element in node.elts: + shape = _static_string_shape(element, constants) + if shape is None: + return None + items.append(shape) + return items + return None + + def _pandas_expression_source(node: ast.AST) -> str | None: """Resolve guarded column-shaped tokens in a static pandas expression.""" @@ -344,11 +401,7 @@ def _call_argument( if len(node.args) > position: return node.args[position] return next( - ( - candidate.value - for candidate in node.keywords - if candidate.arg == keyword - ), + (candidate.value for candidate in node.keywords if candidate.arg == keyword), None, ) @@ -367,6 +420,7 @@ class _SourceReadVisitor(ast.NodeVisitor): def __init__(self, factory_aliases: set[str]) -> None: self.factory_aliases = factory_aliases self.bindings: list[dict[str, str | None]] = [{}] + self.constants: list[dict[str, object]] = [{}] self.accesses: set[tuple[int, int, str]] = set() def _expression(self, node: ast.AST) -> str | None: @@ -381,9 +435,16 @@ def _record(self, node: ast.AST, description: str) -> None: def _bind(self, targets: list[ast.AST], value: ast.AST) -> None: description = self._expression(value) + constant: object = _static_string_shape(value, self.constants) + if constant is None: + constant = _static_string_list(value, self.constants) for target in targets: for name in _assigned_names(target): self.bindings[-1][name] = description + if constant is not None: + self.constants[-1][name] = constant + else: + self.constants[-1].pop(name, None) def visit_Assign(self, node: ast.Assign) -> None: self.visit(node.value) @@ -423,8 +484,10 @@ def _visit_function( if node.args.kwarg is not None: local[node.args.kwarg.arg] = None self.bindings.append(local) + self.constants.append({}) for statement in node.body: self.visit(statement) + self.constants.pop() self.bindings.pop() def visit_FunctionDef(self, node: ast.FunctionDef) -> None: @@ -443,7 +506,9 @@ def visit_Lambda(self, node: ast.Lambda) -> None: ) } self.bindings.append(local) + self.constants.append({}) self.visit(node.body) + self.constants.pop() self.bindings.pop() def visit_Call(self, node: ast.Call) -> None: @@ -467,19 +532,50 @@ def visit_Call(self, node: ast.Call) -> None: elif isinstance(node.func, ast.Attribute) and name in {"query", "eval"}: expression = _call_argument(node, position=0, keyword="expr") if expression is not None: - column = _pandas_expression_source(expression) - if column is not None: - self._record(node, f".{name}() using {column}") + shape = _static_string_shape(expression, self.constants) + if shape is None: + # Fail closed: an expression the guard cannot resolve + # statically could name a guarded column — opacity is + # a violation, not a pass (sol #583 round 2). + self._record( + node, + f".{name}() with an unresolvable dynamic expression " + "(fail-closed)", + ) + else: + for token in re.findall(r"[\w*?\[\]-]+", shape): + if _is_source_column_shape(token): + self._record( + node, + f".{name}() using source column {token!r}", + ) + break elif isinstance(node.func, ast.Attribute) and name == "filter": items = _call_argument(node, position=0, keyword="items") if items is not None: - column = _subscript_source_expression( - items, - bindings=self.bindings, - factory_aliases=self.factory_aliases, - ) - if column is not None: - self._record(node, f".filter(items=...) using {column}") + resolved = _static_string_list(items, self.constants) + if resolved is None: + column = _subscript_source_expression( + items, + bindings=self.bindings, + factory_aliases=self.factory_aliases, + ) + if column is not None: + self._record(node, f".filter(items=...) using {column}") + else: + self._record( + node, + ".filter(items=...) with an unresolvable dynamic " + "list (fail-closed)", + ) + else: + for item in resolved: + if _is_source_column_shape(item): + self._record( + node, + f".filter(items=...) using source column {item!r}", + ) + break self.generic_visit(node) def visit_Attribute(self, node: ast.Attribute) -> None: @@ -875,6 +971,48 @@ def op(df): assert _source_spine_accesses(benign_reads) == () +def test_guard_sees_through_static_indirection_and_fails_closed_on_opacity(): + """Sol #583 round-2 bypasses: one level of static indirection must + resolve, and anything still opaque is a violation by default.""" + + bound_query = """ +def f(df): + expr = 'person_support_channel == "acs"' + return df.query(expr) +""" + bound_eval = """ +def f(df): + expr = 'person_support_channel == "acs"' + return df.eval(expr) +""" + bound_filter = """ +def f(df): + cols = ["person_support_channel"] + return df.filter(items=cols) +""" + fstring_query = """ +def f(df): + col = "person_support_channel" + return df.query(f"{col} == 'acs'") +""" + opaque_query = """ +def f(df, expr): + return df.query(expr) +""" + benign_bound = """ +def f(df): + expr = "age >= 18" + return df.query(expr) +""" + assert _source_spine_accesses(bound_query) + assert _source_spine_accesses(bound_eval) + assert _source_spine_accesses(bound_filter) + assert _source_spine_accesses(fstring_query) + accesses = _source_spine_accesses(opaque_query) + assert accesses and any("fail-closed" in access for access in accesses) + assert not _source_spine_accesses(benign_bound) + + def test_source_spine_ast_guard_covers_every_entity_grain() -> None: """Every US and benefit-unit grain's source identity is prohibited.""" From 19a7a1d4cf05c2a3fedc1f22b7cce04b5ebcc902 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 23:39:18 -0400 Subject: [PATCH 35/94] Round 3: wildcards, hidden args, shadows, and mutations are all opaque MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sol round-3 evasions closed at the root: an unresolvable interpolation produced a '*' wildcard that read as resolved — wildcards now mean opaque at every strict call site, alongside hidden/expanded call arguments (**kwargs), shadowing parameters and conditional reassigns (explicit None shadows that never fall through to outer constants), and constant staleness from list/dict mutation (mutator calls, subscript and augmented assignment poison the binding). str.format resolves through the constants scope so a fully-static template still reports the column by name rather than opacity. Under the completed rule nothing survives silently: every pandas expression is resolved-and- checked or fail-closed. The 54-module operator graph scans clean; all six round-3 evasions are committed self-tests. Co-Authored-By: Claude Fable 5 --- .../tests/test_us_spine_blindness.py | 136 ++++++++++++++++-- 1 file changed, 127 insertions(+), 9 deletions(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index b0d411cd..a0c778f3 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -341,6 +341,8 @@ def _static_string_shape( value = scope[node.id] return value if isinstance(value, str) else None return None + if isinstance(node, ast.IfExp): + return None if isinstance(node, ast.FormattedValue): inner = _static_string_shape(node.value, constants) return inner if inner is not None else "*" @@ -355,6 +357,19 @@ def _static_string_shape( if left is not None and right is not None: return left + right return None + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and node.func.attr == "format" + and not node.keywords + ): + template = _static_string_shape(node.func.value, constants) + if template is None: + return None + for argument in node.args: + value = _static_string_shape(argument, constants) + template = template.replace("{}", "*" if value is None else value, 1) + return template return _string_shape(node) @@ -441,15 +456,23 @@ def _bind(self, targets: list[ast.AST], value: ast.AST) -> None: for target in targets: for name in _assigned_names(target): self.bindings[-1][name] = description - if constant is not None: - self.constants[-1][name] = constant - else: - self.constants[-1].pop(name, None) + # None is an explicit opaque shadow: lookups must stop here, + # never fall through to a stale outer binding (shadowed + # parameters and conditional reassignments — sol round 3). + self.constants[-1][name] = constant def visit_Assign(self, node: ast.Assign) -> None: self.visit(node.value) + for target in node.targets: + if isinstance(target, ast.Subscript) and isinstance(target.value, ast.Name): + self._poison(target.value.id) self._bind(list(node.targets), node.value) + def visit_AugAssign(self, node: ast.AugAssign) -> None: + self.visit(node.value) + for name in _assigned_names(node.target): + self._poison(name) + def visit_AnnAssign(self, node: ast.AnnAssign) -> None: if node.value is None: for name in _assigned_names(node.target): @@ -484,7 +507,7 @@ def _visit_function( if node.args.kwarg is not None: local[node.args.kwarg.arg] = None self.bindings.append(local) - self.constants.append({}) + self.constants.append({name: None for name in local}) for statement in node.body: self.visit(statement) self.constants.pop() @@ -506,12 +529,36 @@ def visit_Lambda(self, node: ast.Lambda) -> None: ) } self.bindings.append(local) - self.constants.append({}) + self.constants.append({name: None for name in local}) self.visit(node.body) self.constants.pop() self.bindings.pop() + _MUTATORS = frozenset( + { + "append", + "extend", + "insert", + "remove", + "clear", + "pop", + "sort", + "reverse", + "update", + "setdefault", + } + ) + + def _poison(self, name: str) -> None: + self.constants[-1][name] = None + def visit_Call(self, node: ast.Call) -> None: + if ( + isinstance(node.func, ast.Attribute) + and isinstance(node.func.value, ast.Name) + and node.func.attr in self._MUTATORS + ): + self._poison(node.func.value.id) name = _call_name(node) if name in self.factory_aliases: self._record(node, f"call to {name}") @@ -531,9 +578,14 @@ def visit_Call(self, node: ast.Call) -> None: self._record(node, f".get() using {column}") elif isinstance(node.func, ast.Attribute) and name in {"query", "eval"}: expression = _call_argument(node, position=0, keyword="expr") - if expression is not None: + if expression is None and (node.args or node.keywords): + self._record( + node, + f".{name}() with hidden or expanded arguments (fail-closed)", + ) + elif expression is not None: shape = _static_string_shape(expression, self.constants) - if shape is None: + if shape is None or "*" in shape: # Fail closed: an expression the guard cannot resolve # statically could name a guarded column — opacity is # a violation, not a pass (sol #583 round 2). @@ -552,7 +604,12 @@ def visit_Call(self, node: ast.Call) -> None: break elif isinstance(node.func, ast.Attribute) and name == "filter": items = _call_argument(node, position=0, keyword="items") - if items is not None: + if items is None and (node.args or node.keywords): + self._record( + node, + ".filter() with hidden or expanded arguments (fail-closed)", + ) + elif items is not None: resolved = _static_string_list(items, self.constants) if resolved is None: column = _subscript_source_expression( @@ -568,6 +625,12 @@ def visit_Call(self, node: ast.Call) -> None: ".filter(items=...) with an unresolvable dynamic " "list (fail-closed)", ) + elif any("*" in item for item in resolved): + self._record( + node, + ".filter(items=...) with an unresolvable dynamic " + "list (fail-closed)", + ) else: for item in resolved: if _is_source_column_shape(item): @@ -1013,6 +1076,61 @@ def f(df): assert not _source_spine_accesses(benign_bound) +def test_guard_treats_wildcards_hidden_args_and_mutations_as_opaque(): + """Sol #583 round-3 evasions: str.format and parameter interpolation + resolved to a '*' wildcard and read as benign; kwargs-expansion hid + the expression; mutation staled a resolved list; a shadowing + parameter fell through to an outer constant. All are opaque now.""" + + format_bound = """ +def f(df): + col = "person_support_channel" + return df.query("{} == 'acs'".format(col)) +""" + param_fstring = """ +def f(df, col): + return df.query(f"{col} == 'acs'") +""" + kwargs_hidden = """ +def f(df, kw): + return df.query(**kw) +""" + mutated_filter = """ +def f(df): + cols = ["age"] + cols.append("person_support_channel") + return df.filter(items=cols) +""" + conditional_bound = """ +def f(df, flag): + col = "age" if flag else "person_support_channel" + return df.query(f"{col} >= 1") +""" + shadowed_param = """ +col = "person_support_channel" + + +def f(df, col): + return df.query(f"{col} >= 1") +""" + for source in ( + format_bound, + param_fstring, + kwargs_hidden, + mutated_filter, + conditional_bound, + shadowed_param, + ): + accesses = _source_spine_accesses(source) + assert accesses, source + # Precision is preserved where resolution succeeds: format_bound's + # template resolves fully, so it reports the column, not opacity. + assert any( + "person_support_channel" in access + for access in _source_spine_accesses(format_bound) + ) + + def test_source_spine_ast_guard_covers_every_entity_grain() -> None: """Every US and benefit-unit grain's source identity is prohibited.""" From 4fa8c11d5d816cdc85f787caf9c424570c589ca4 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 30 Jul 2026 23:51:13 -0400 Subject: [PATCH 36/94] Start PR 583 round-4 guard journal --- PROGRESS.md | 64 +++++++++++++++++++++-------------------------------- 1 file changed, 25 insertions(+), 39 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index f3158fdd..1f2336c1 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,48 +1,34 @@ -# Progress +# PR #583 Guard Completion Progress ## State -Populace #516 whole-row donor outlier screen is complete on -`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515 -interim carve merged as #525). The `puf_tax_detail` donor now drops tax units -whose grouped raw mortgage interest reaches $10M before the #515 carve -(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T -of phantom mortgage-interest mass), with the checkpoint schema bumped to v3 -so post-carve pre-screen checkpoints rebuild. +Round-4 hold remediation is in progress on `multispine-pool-build-578`, from +clean starting commit `19a7a1d`. The operator graph is clean at 54 modules, but +the guard in +`packages/populace-build/tests/test_us_spine_blindness.py` is still fail-open +for unresolved subscripts, incomplete `str.format` fields, method aliases, and +late-bound closure values. + +No push or external mutation is authorized. ## Done -- Confirmed a clean starting worktree at `aef1c56`. -- Read the repository guidance and established the #515 donor carve as the - screen's required downstream boundary. -- Started source-level audits of every donor-frame consumer, checkpoint - validation, row-count pins, and existing donor-fact summaries. -- Attempted the requested GitNexus impact workflow; the managed filesystem - denied its global registry write. Its local index also exposed a broad - `build/` ignore mismatch, so the completed impact audit uses direct source - call sites and tests. -- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the - structural rationale and pinned-artifact receipts. -- Added a whole-row screen on grouped raw person `home_mortgage_interest` - after tax-unit assembly, before the #515 carve, with retained-index reset. -- Confirmed no downstream consumer pairs donor rows to the original HDF arrays - or carries a stale donor-length vector; values and weights always originate - from the same screened frame. -- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale - checkpoint regression track the live constant while retaining literal-v1 - corruptions. -- Added regression coverage for the exact grouped boundary, whole-row removal, - retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant. -- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets - 139; populace-data 138 with 1 skip. The directly affected tail-bound suite - adds 12 passes. Ruff format/check and `git diff --check` are clean. -- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line - audit, expected 208,611-row real-artifact effect, verification results, count - sweep, and deliberately untouched surfaces. +- Read `CLAUDE.md` and the round-4 review log. +- Confirmed the required clean starting HEAD and branch. +- Confirmed the local GitNexus index is absent; direct AST/source tracing will + be used unless a safe local index becomes available. +- Started parallel read-only audits of reviewer repros, resolver/loop behavior, + and strict call-site/closure behavior. ## Next -- PR #527 review cycle, then merge. After both #525 and #527: rebuild the - base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a - run that holds per `us_critical_targets.py`. -- Root record-level ETL carve stays open on populace#515. +- Commit this baseline journal. +- Add committed self-tests and implementation commits for, in order: + subscripts/resolver completion; loop/comprehension propagation; complete + format fields; method aliasing; closure late binding. +- Add the cross-round completeness invariant and benign battery. +- Run the guard file, the full `populace-build` suite, Ruff, and the 54-module + graph scan. +- Write `/private/tmp/583_fix2_handoff.md` with per-subsystem rules, binding + tests, any operator rewrite rationale, exact validation results, and commit + inventory. From 972a22b197352178c27828e5ff8c11de1ad9cb2b Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 00:05:36 -0400 Subject: [PATCH 37/94] Fail closed on opaque column subscripts --- PROGRESS.md | 28 +- .../tests/test_us_spine_blindness.py | 302 +++++++++++++++++- 2 files changed, 306 insertions(+), 24 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 1f2336c1..da6563d7 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -3,11 +3,10 @@ ## State Round-4 hold remediation is in progress on `multispine-pool-build-578`, from -clean starting commit `19a7a1d`. The operator graph is clean at 54 modules, but -the guard in -`packages/populace-build/tests/test_us_spine_blindness.py` is still fail-open -for unresolved subscripts, incomplete `str.format` fields, method aliases, and -late-bound closure values. +clean starting commit `19a7a1d`. Subscript resolution now catches all six +round-4 evasions, and the 54-module operator graph remains clean. Loop and +comprehension binding, complete `str.format` fields, method aliases, and +late-bound closure values remain in progress. No push or external mutation is authorized. @@ -17,15 +16,24 @@ No push or external mutation is authorized. - Confirmed the required clean starting HEAD and branch. - Confirmed the local GitNexus index is absent; direct AST/source tracing will be used unless a safe local index becomes available. -- Started parallel read-only audits of reviewer repros, resolver/loop behavior, - and strict call-site/closure behavior. +- Completed parallel read-only audits of reviewer repros, resolver/loop + behavior, and strict call-site/closure behavior. +- Extended the static resolver for walrus expressions, string multiplication, + percent formatting, and all-static chained `str.replace`. +- Made inferred column-container subscripts resolve every static string member + or record explicit opacity; subscript assignment targets are visited too. +- Added exact round-4 binding tests: walrus binds and reports both reads; + multiplication, percent formatting, and replace report the guarded name; + nested calls and dict indirection report fail-closed. +- Replayed the guard file (15 passed), focused Ruff, and the exact 54-module + graph (no missing modules or offenders). ## Next -- Commit this baseline journal. +- Commit the subscript/resolver completion. - Add committed self-tests and implementation commits for, in order: - subscripts/resolver completion; loop/comprehension propagation; complete - format fields; method aliasing; closure late binding. + loop/comprehension propagation; complete format fields; method aliasing; + closure late binding. - Add the cross-round completeness invariant and benign battery. - Run the guard file, the full `populace-build` suite, Ruff, and the 54-module graph scan. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index a0c778f3..36bfc823 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -341,6 +341,8 @@ def _static_string_shape( value = scope[node.id] return value if isinstance(value, str) else None return None + if isinstance(node, ast.NamedExpr): + return _static_string_shape(node.value, constants) if isinstance(node, ast.IfExp): return None if isinstance(node, ast.FormattedValue): @@ -351,12 +353,53 @@ def _static_string_shape( if any(piece is None for piece in pieces): return None return "".join(piece for piece in pieces if piece is not None) - if isinstance(node, ast.BinOp) and isinstance(node.op, ast.Add): - left = _static_string_shape(node.left, constants) - right = _static_string_shape(node.right, constants) - if left is not None and right is not None: - return left + right - return None + if isinstance(node, ast.BinOp): + if isinstance(node.op, ast.Add): + left = _static_string_shape(node.left, constants) + right = _static_string_shape(node.right, constants) + if left is not None and right is not None: + return left + right + return None + if isinstance(node.op, ast.Mult): + left_string = _static_string_shape(node.left, constants) + right_string = _static_string_shape(node.right, constants) + left_integer = _static_integer(node.left, constants) + right_integer = _static_integer(node.right, constants) + if left_string is not None and right_integer is not None: + return left_string * right_integer + if left_integer is not None and right_string is not None: + return left_integer * right_string + return None + if isinstance(node.op, ast.Mod): + template = _static_string_shape(node.left, constants) + operand = _static_percent_operand(node.right, constants) + if template is None or operand is _OPAQUE_STATIC_VALUE: + return None + try: + result = template % operand + except (TypeError, ValueError): + return None + return result if isinstance(result, str) else None + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and node.func.attr == "replace" + and not node.keywords + and len(node.args) in {2, 3} + ): + value = _static_string_shape(node.func.value, constants) + old = _static_string_shape(node.args[0], constants) + new = _static_string_shape(node.args[1], constants) + count = ( + _static_integer(node.args[2], constants) if len(node.args) == 3 else None + ) + if value is None or old is None or new is None: + return None + if len(node.args) == 3 and count is None: + return None + return ( + value.replace(old, new) if count is None else value.replace(old, new, count) + ) if ( isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) @@ -373,16 +416,64 @@ def _static_string_shape( return _string_shape(node) +_OPAQUE_STATIC_VALUE = object() + + +def _static_integer(node: ast.AST, constants: list[dict[str, object]]) -> int | None: + if isinstance(node, ast.Constant) and isinstance(node.value, int): + return node.value + if isinstance(node, ast.Name): + for scope in reversed(constants): + if node.id in scope: + value = scope[node.id] + return value if isinstance(value, int) else None + if isinstance(node, ast.NamedExpr): + return _static_integer(node.value, constants) + return None + + +def _static_percent_operand( + node: ast.AST, + constants: list[dict[str, object]], +) -> object: + if isinstance(node, ast.Constant) and isinstance( + node.value, (str, int, float, bytes) + ): + return node.value + if isinstance(node, ast.Name): + for scope in reversed(constants): + if node.id in scope: + value = scope[node.id] + if isinstance(value, (str, int, float, bytes, tuple)): + return value + return _OPAQUE_STATIC_VALUE + return _OPAQUE_STATIC_VALUE + if isinstance(node, ast.NamedExpr): + return _static_percent_operand(node.value, constants) + if isinstance(node, ast.Tuple): + values = tuple( + _static_percent_operand(element, constants) for element in node.elts + ) + if any(value is _OPAQUE_STATIC_VALUE for value in values): + return _OPAQUE_STATIC_VALUE + return values + return _OPAQUE_STATIC_VALUE + + def _static_string_list( node: ast.AST, constants: list[dict[str, object]] -) -> list[str] | None: +) -> tuple[str, ...] | None: """Resolve a static list/tuple of strings, through one Name binding.""" if isinstance(node, ast.Name): for scope in reversed(constants): if node.id in scope: value = scope[node.id] - return value if isinstance(value, list) else None + if isinstance(value, tuple) and all( + isinstance(item, str) for item in value + ): + return value + return None return None if isinstance(node, (ast.List, ast.Tuple)): items: list[str] = [] @@ -391,10 +482,20 @@ def _static_string_list( if shape is None: return None items.append(shape) - return items + return tuple(items) return None +def _static_string_values( + node: ast.AST, + constants: list[dict[str, object]], +) -> tuple[str, ...] | None: + shape = _static_string_shape(node, constants) + if shape is not None: + return (shape,) + return _static_string_list(node, constants) + + def _pandas_expression_source(node: ast.AST) -> str | None: """Resolve guarded column-shaped tokens in a static pandas expression.""" @@ -436,6 +537,7 @@ def __init__(self, factory_aliases: set[str]) -> None: self.factory_aliases = factory_aliases self.bindings: list[dict[str, str | None]] = [{}] self.constants: list[dict[str, object]] = [{}] + self.column_containers: list[dict[str, bool]] = [{}] self.accesses: set[tuple[int, int, str]] = set() def _expression(self, node: ast.AST) -> str | None: @@ -448,11 +550,28 @@ def _expression(self, node: ast.AST) -> str | None: def _record(self, node: ast.AST, description: str) -> None: self.accesses.add((node.lineno, node.col_offset, description)) + def _column_container(self, node: ast.AST) -> bool: + if isinstance(node, ast.Name): + for scope in reversed(self.column_containers): + if node.id in scope: + return scope[node.id] + return False + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and node.func.attr == "copy" + ): + return self._column_container(node.func.value) + return False + def _bind(self, targets: list[ast.AST], value: ast.AST) -> None: description = self._expression(value) constant: object = _static_string_shape(value, self.constants) if constant is None: constant = _static_string_list(value, self.constants) + if constant is None: + constant = _static_integer(value, self.constants) + container = self._column_container(value) for target in targets: for name in _assigned_names(target): self.bindings[-1][name] = description @@ -460,16 +579,20 @@ def _bind(self, targets: list[ast.AST], value: ast.AST) -> None: # never fall through to a stale outer binding (shadowed # parameters and conditional reassignments — sol round 3). self.constants[-1][name] = constant + self.column_containers[-1][name] = container def visit_Assign(self, node: ast.Assign) -> None: self.visit(node.value) for target in node.targets: - if isinstance(target, ast.Subscript) and isinstance(target.value, ast.Name): - self._poison(target.value.id) + if isinstance(target, ast.Subscript): + self.visit(target) + if isinstance(target.value, ast.Name): + self._poison(target.value.id) self._bind(list(node.targets), node.value) def visit_AugAssign(self, node: ast.AugAssign) -> None: self.visit(node.value) + self.visit(node.target) for name in _assigned_names(node.target): self._poison(name) @@ -477,8 +600,12 @@ def visit_AnnAssign(self, node: ast.AnnAssign) -> None: if node.value is None: for name in _assigned_names(node.target): self.bindings[-1][name] = None + self.constants[-1][name] = None + self.column_containers[-1][name] = False return self.visit(node.value) + if isinstance(node.target, ast.Subscript): + self.visit(node.target) self._bind([node.target], node.value) def visit_NamedExpr(self, node: ast.NamedExpr) -> None: @@ -506,10 +633,26 @@ def _visit_function( local[node.args.vararg.arg] = None if node.args.kwarg is not None: local[node.args.kwarg.arg] = None + arguments = ( + *node.args.posonlyargs, + *node.args.args, + *node.args.kwonlyargs, + ) + containers = { + argument.arg: argument.annotation is None + and argument.arg not in {"self", "cls"} + for argument in arguments + } + if node.args.vararg is not None: + containers[node.args.vararg.arg] = False + if node.args.kwarg is not None: + containers[node.args.kwarg.arg] = False self.bindings.append(local) self.constants.append({name: None for name in local}) + self.column_containers.append(containers) for statement in node.body: self.visit(statement) + self.column_containers.pop() self.constants.pop() self.bindings.pop() @@ -528,9 +671,20 @@ def visit_Lambda(self, node: ast.Lambda) -> None: *node.args.kwonlyargs, ) } + containers = { + argument.arg: argument.annotation is None + and argument.arg not in {"self", "cls"} + for argument in ( + *node.args.posonlyargs, + *node.args.args, + *node.args.kwonlyargs, + ) + } self.bindings.append(local) self.constants.append({name: None for name in local}) + self.column_containers.append(containers) self.visit(node.body) + self.column_containers.pop() self.constants.pop() self.bindings.pop() @@ -647,13 +801,46 @@ def visit_Attribute(self, node: ast.Attribute) -> None: self.generic_visit(node) def visit_Subscript(self, node: ast.Subscript) -> None: + selector = node.slice + column_container = self._column_container(node.value) + if ( + isinstance(node.value, ast.Attribute) + and node.value.attr == "loc" + and isinstance(node.slice, ast.Tuple) + and node.slice.elts + ): + selector = node.slice.elts[-1] + column_container = self._column_container(node.value.value) + + resolved = _static_string_values(selector, self.constants) column = _subscript_source_expression( - node.slice, + selector, bindings=self.bindings, factory_aliases=self.factory_aliases, ) - if column is not None: - self._record(node, f"subscript using {column}") + if resolved is None: + if column is not None: + self._record(node, f"subscript using {column}") + elif column_container: + self._record( + node, + "subscript with an unresolvable dynamic selector (fail-closed)", + ) + elif any("*" in item for item in resolved): + if column_container or any( + _is_source_column_shape(item) for item in resolved + ): + self._record( + node, + "subscript with an unresolvable dynamic selector (fail-closed)", + ) + else: + for item in resolved: + if _is_source_column_shape(item): + self._record( + node, + f"subscript using source column {item!r}", + ) self.generic_visit(node) @@ -1131,6 +1318,93 @@ def f(df, col): ) +def test_subscript_selectors_resolve_or_fail_closed() -> None: + """Every round-4 subscript evasion is named or explicitly opaque.""" + + walrus = """ +def f(df): + df[(col := "person_support_channel")] + return df[col] +""" + nested_call = """ +def f(df): + def source_column(): + return "person_support_channel" + return df[source_column()] +""" + dict_indirection = """ +COLS = {"unsafe": "person_support_channel"} +def f(df): + return df[COLS["unsafe"]] +""" + multiplication = """ +def f(df): + return df["person_support_channel" * 1] +""" + percent_format = """ +def f(df): + return df["%s_support_channel" % "person"] +""" + replace_chain = """ +def f(df): + return df[ + "person-SOURCE-channel" + .replace("-", "_") + .replace("SOURCE", "support") + ] +""" + + walrus_accesses = _source_spine_accesses(walrus) + assert len(walrus_accesses) == 2 + assert all("person_support_channel" in access for access in walrus_accesses) + + for source in (multiplication, percent_format, replace_chain): + accesses = _source_spine_accesses(source) + assert accesses, source + assert all("person_support_channel" in access for access in accesses) + assert all("fail-closed" not in access for access in accesses) + + for source in (nested_call, dict_indirection): + accesses = _source_spine_accesses(source) + assert accesses, source + assert any("fail-closed" in access for access in accesses) + + +def test_static_selector_extensions_are_shared_by_strict_pandas_calls() -> None: + """Mult, percent formatting, and replace resolve at every strict surface.""" + + sources = ( + """ +def f(df): + return df.query("person_support_channel == 1" * 1) +""", + """ +def f(df): + return df.eval("%s_support_channel == 1" % "person") +""", + """ +def f(df): + return df.query( + "person_x == 1" + .replace("x", "support") + .replace("support", "support_channel") + ) +""", + """ +def f(df): + return df.query( + (expr := "person_support_channel == 1") + ) +""", + ) + + for source in sources: + accesses = _source_spine_accesses(source) + assert accesses, source + assert all("person_support_channel" in access for access in accesses) + assert all("fail-closed" not in access for access in accesses) + + def test_source_spine_ast_guard_covers_every_entity_grain() -> None: """Every US and benefit-unit grain's source identity is prohibited.""" From d836881151098d709b4c17ce74b4566cda513fcb Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 00:07:21 -0400 Subject: [PATCH 38/94] Propagate static column choices through iteration --- PROGRESS.md | 21 ++- .../tests/test_us_spine_blindness.py | 162 +++++++++++++++++- 2 files changed, 173 insertions(+), 10 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index da6563d7..5203bbce 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -4,9 +4,10 @@ Round-4 hold remediation is in progress on `multispine-pool-build-578`, from clean starting commit `19a7a1d`. Subscript resolution now catches all six -round-4 evasions, and the 54-module operator graph remains clean. Loop and -comprehension binding, complete `str.format` fields, method aliases, and -late-bound closure values remain in progress. +round-4 evasions; loop and comprehension targets now propagate every static +string choice or an opaque shadow; and the 54-module operator graph remains +clean. Complete `str.format` fields, method aliases, and late-bound closure +values remain in progress. No push or external mutation is authorized. @@ -27,13 +28,19 @@ No push or external mutation is authorized. nested calls and dict indirection report fail-closed. - Replayed the guard file (15 passed), focused Ruff, and the exact 54-module graph (no missing modules or offenders). +- Bound `for`/`async for` targets and list, set, dict, and generator + comprehension targets before their bodies are visited. +- Represented statically resolvable iterables as tuples of every possible + string, checking every member at a subscript; dynamic iterables explicitly + shadow stale outer bindings with opacity. +- Added benign, mixed guarded, module-bound, dynamic, and comprehension-wrapped + selector regressions. The guard file now passes 17 tests with the graph clean. ## Next -- Commit the subscript/resolver completion. -- Add committed self-tests and implementation commits for, in order: - loop/comprehension propagation; complete format fields; method aliasing; - closure late binding. +- Commit loop/comprehension propagation. +- Add committed self-tests and implementation commits for complete format + fields, method aliasing, and closure late binding. - Add the cross-round completeness invariant and benign battery. - Run the guard file, the full `populace-build` suite, Ruff, and the 54-module graph scan. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 36bfc823..cedbefa6 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -478,11 +478,23 @@ def _static_string_list( if isinstance(node, (ast.List, ast.Tuple)): items: list[str] = [] for element in node.elts: - shape = _static_string_shape(element, constants) - if shape is None: + values = _static_string_values(element, constants) + if values is None: return None - items.append(shape) + items.extend(values) return tuple(items) + if isinstance(node, (ast.ListComp, ast.SetComp, ast.GeneratorExp)): + local: dict[str, object] = {} + nested_constants = [*constants, local] + for generator in node.generators: + if generator.ifs or generator.is_async: + return None + values = _static_string_list(generator.iter, nested_constants) + if values is None: + return None + for name in _assigned_names(generator.target): + local[name] = values + return _static_string_values(node.elt, nested_constants) return None @@ -612,6 +624,64 @@ def visit_NamedExpr(self, node: ast.NamedExpr) -> None: self.visit(node.value) self._bind([node.target], node.value) + def _bind_iteration_target( + self, + target: ast.AST, + values: tuple[str, ...] | None, + ) -> None: + for name in _assigned_names(target): + self.bindings[-1][name] = None + self.constants[-1][name] = values + self.column_containers[-1][name] = False + + def visit_For(self, node: ast.For) -> None: + self.visit(node.iter) + self._bind_iteration_target( + node.target, + _static_string_list(node.iter, self.constants), + ) + for statement in (*node.body, *node.orelse): + self.visit(statement) + + def visit_AsyncFor(self, node: ast.AsyncFor) -> None: + self.visit_For(node) + + def _visit_comprehension( + self, + node: ast.ListComp | ast.SetComp | ast.DictComp | ast.GeneratorExp, + ) -> None: + self.bindings.append({}) + self.constants.append({}) + self.column_containers.append({}) + for generator in node.generators: + self.visit(generator.iter) + self._bind_iteration_target( + generator.target, + _static_string_list(generator.iter, self.constants), + ) + for condition in generator.ifs: + self.visit(condition) + if isinstance(node, ast.DictComp): + self.visit(node.key) + self.visit(node.value) + else: + self.visit(node.elt) + self.column_containers.pop() + self.constants.pop() + self.bindings.pop() + + def visit_ListComp(self, node: ast.ListComp) -> None: + self._visit_comprehension(node) + + def visit_SetComp(self, node: ast.SetComp) -> None: + self._visit_comprehension(node) + + def visit_DictComp(self, node: ast.DictComp) -> None: + self._visit_comprehension(node) + + def visit_GeneratorExp(self, node: ast.GeneratorExp) -> None: + self._visit_comprehension(node) + def _visit_function( self, node: ast.FunctionDef | ast.AsyncFunctionDef, @@ -1405,6 +1475,92 @@ def f(df): assert all("fail-closed" not in access for access in accesses) +def test_loop_targets_propagate_every_static_string_choice() -> None: + """Loop selectors check every static member and shadow dynamic iterables.""" + + benign = """ +def f(df): + for col in ("age", "income"): + df[col] +""" + guarded = """ +COLUMNS = ("age", "person_support_channel", "household_spine") +def f(df): + for col in COLUMNS: + df[col] +""" + dynamic = """ +col = "age" +def f(df, columns): + for col in columns: + df[col] +""" + + assert _source_spine_accesses(benign) == () + guarded_accesses = _source_spine_accesses(guarded) + assert len(guarded_accesses) == 2 + assert any("person_support_channel" in item for item in guarded_accesses) + assert any("household_spine" in item for item in guarded_accesses) + dynamic_accesses = _source_spine_accesses(dynamic) + assert dynamic_accesses + assert all("fail-closed" in item for item in dynamic_accesses) + + +def test_comprehension_targets_propagate_static_and_opaque_choices() -> None: + """All comprehension forms bind targets before visiting their bodies.""" + + benign_sources = ( + """ +def f(df): + return [df[col] for col in ("age", "income")] +""", + """ +def f(df): + return {df[col] for col in ("age", "income")} +""", + """ +def f(df): + return {col: df[col] for col in ("age", "income")} +""", + """ +def f(df): + return tuple(df[col] for col in ("age", "income")) +""", + """ +def f(df): + return df[[col for col in ("age", "income")]] +""", + ) + guarded_sources = tuple( + source.replace( + '("age", "income")', + '("age", "person_support_channel")', + ) + for source in benign_sources + ) + dynamic_sources = tuple( + source.replace( + "def f(df):", + "def f(df, columns):", + ).replace( + '("age", "income")', + "columns", + ) + for source in benign_sources + ) + + for source in benign_sources: + assert _source_spine_accesses(source) == (), source + for source in guarded_sources: + accesses = _source_spine_accesses(source) + assert accesses, source + assert any("person_support_channel" in item for item in accesses) + for source in dynamic_sources: + accesses = _source_spine_accesses(source) + assert accesses, source + assert any("fail-closed" in item for item in accesses) + + def test_source_spine_ast_guard_covers_every_entity_grain() -> None: """Every US and benefit-unit grain's source identity is prohibited.""" From a88fef1b5f2054188f3fd7b793b0d508ec0fb7ad Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 00:09:26 -0400 Subject: [PATCH 39/94] Resolve complete static format fields --- PROGRESS.md | 18 +- .../tests/test_us_spine_blindness.py | 208 +++++++++++++++++- 2 files changed, 210 insertions(+), 16 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 5203bbce..0f32bfad 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -6,8 +6,8 @@ Round-4 hold remediation is in progress on `multispine-pool-build-578`, from clean starting commit `19a7a1d`. Subscript resolution now catches all six round-4 evasions; loop and comprehension targets now propagate every static string choice or an opaque shadow; and the 54-module operator graph remains -clean. Complete `str.format` fields, method aliases, and late-bound closure -values remain in progress. +clean. `str.format` now resolves all requested static field forms precisely. +Method aliases and late-bound closure values remain in progress. No push or external mutation is authorized. @@ -35,12 +35,20 @@ No push or external mutation is authorized. shadow stale outer bindings with opacity. - Added benign, mixed guarded, module-bound, dynamic, and comprehension-wrapped selector regressions. The guard file now passes 17 tests with the graph clean. +- Replaced bare `{}` substitution with `string.Formatter` field parsing for + automatic, indexed, named, converted (`!s`/`!r`), and specified fields. +- Kept unresolved fields structurally opaque while preserving literal `*` in a + fully static pandas expression as benign syntax. +- Bound exact-name checks for every unsafe format variant and zero-finding + controls for named benign formatting, conversions/specs, escaped braces, and + multiplication syntax. The guard file now passes 19 tests with the graph + clean. ## Next -- Commit loop/comprehension propagation. -- Add committed self-tests and implementation commits for complete format - fields, method aliasing, and closure late binding. +- Commit complete format-field resolution. +- Add committed self-tests and implementation commits for method aliasing and + closure late binding. - Add the cross-round completeness invariant and benign battery. - Run the guard file, the full `populace-build` suite, Ruff, and the 54-module graph scan. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index cedbefa6..f9e2a54f 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -6,6 +6,7 @@ import fnmatch import re from pathlib import Path +from string import Formatter _REPOSITORY_ROOT = Path(__file__).resolve().parents[3] _US_RUNTIME = ( @@ -347,7 +348,7 @@ def _static_string_shape( return None if isinstance(node, ast.FormattedValue): inner = _static_string_shape(node.value, constants) - return inner if inner is not None else "*" + return inner if inner is not None else _OPAQUE_STRING_PART if isinstance(node, ast.JoinedStr): pieces = [_static_string_shape(value, constants) for value in node.values] if any(piece is None for piece in pieces): @@ -380,6 +381,18 @@ def _static_string_shape( except (TypeError, ValueError): return None return result if isinstance(result, str) else None + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and node.func.attr == "join" + and len(node.args) == 1 + and not node.keywords + ): + separator = _static_string_shape(node.func.value, constants) + values = _static_string_list(node.args[0], constants) + if separator is None or values is None: + return None + return separator.join(values) if ( isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) @@ -404,19 +417,92 @@ def _static_string_shape( isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) and node.func.attr == "format" - and not node.keywords ): template = _static_string_shape(node.func.value, constants) if template is None: return None - for argument in node.args: - value = _static_string_shape(argument, constants) - template = template.replace("{}", "*" if value is None else value, 1) - return template + return _resolve_static_format(template, node, constants) return _string_shape(node) _OPAQUE_STATIC_VALUE = object() +_OPAQUE_STRING_PART = "\N{OBJECT REPLACEMENT CHARACTER}" + + +def _static_format_value( + node: ast.AST, + constants: list[dict[str, object]], +) -> object: + value = _static_string_shape(node, constants) + if value is not None: + return _OPAQUE_STATIC_VALUE if _OPAQUE_STRING_PART in value else value + integer = _static_integer(node, constants) + if integer is not None: + return integer + if isinstance(node, ast.Constant) and isinstance( + node.value, + (float, bool, bytes, type(None)), + ): + return node.value + return _OPAQUE_STATIC_VALUE + + +def _resolve_static_format( + template: str, + node: ast.Call, + constants: list[dict[str, object]], +) -> str: + """Substitute every statically known ``str.format`` field.""" + + formatter = Formatter() + positional = tuple( + _OPAQUE_STATIC_VALUE + if isinstance(argument, ast.Starred) + else _static_format_value(argument, constants) + for argument in node.args + ) + keywords = { + keyword.arg: _static_format_value(keyword.value, constants) + for keyword in node.keywords + if keyword.arg is not None + } + expanded_keywords = any(keyword.arg is None for keyword in node.keywords) + auto_index = 0 + pieces: list[str] = [] + try: + parsed = tuple(formatter.parse(template)) + except ValueError: + return _OPAQUE_STRING_PART + for literal, field_name, format_spec, conversion in parsed: + pieces.append(literal) + if field_name is None: + continue + lookup_name = field_name + if field_name == "": + lookup_name = str(auto_index) + auto_index += 1 + try: + value, _ = formatter.get_field( + lookup_name, + positional, + keywords, + ) + except (AttributeError, IndexError, KeyError, TypeError, ValueError): + value = _OPAQUE_STATIC_VALUE + if value is _OPAQUE_STATIC_VALUE or ( + expanded_keywords and not lookup_name.isdecimal() + ): + pieces.append(_OPAQUE_STRING_PART) + continue + if "{" in format_spec or "}" in format_spec: + pieces.append(_OPAQUE_STRING_PART) + continue + try: + converted = formatter.convert_field(value, conversion) + pieces.append(formatter.format_field(converted, format_spec)) + except (TypeError, ValueError): + pieces.append(_OPAQUE_STRING_PART) + return "".join(pieces) def _static_integer(node: ast.AST, constants: list[dict[str, object]]) -> int | None: @@ -809,7 +895,7 @@ def visit_Call(self, node: ast.Call) -> None: ) elif expression is not None: shape = _static_string_shape(expression, self.constants) - if shape is None or "*" in shape: + if shape is None or _OPAQUE_STRING_PART in shape: # Fail closed: an expression the guard cannot resolve # statically could name a guarded column — opacity is # a violation, not a pass (sol #583 round 2). @@ -849,7 +935,7 @@ def visit_Call(self, node: ast.Call) -> None: ".filter(items=...) with an unresolvable dynamic " "list (fail-closed)", ) - elif any("*" in item for item in resolved): + elif any(_OPAQUE_STRING_PART in item for item in resolved): self._record( node, ".filter(items=...) with an unresolvable dynamic " @@ -896,9 +982,11 @@ def visit_Subscript(self, node: ast.Subscript) -> None: node, "subscript with an unresolvable dynamic selector (fail-closed)", ) - elif any("*" in item for item in resolved): - if column_container or any( - _is_source_column_shape(item) for item in resolved + elif any(_OPAQUE_STRING_PART in item for item in resolved): + if ( + column_container + or column is not None + or any(_is_source_column_shape(item) for item in resolved) ): self._record( node, @@ -1561,6 +1649,104 @@ def f(df): assert any("fail-closed" in item for item in accesses) +def test_format_fields_resolve_all_static_forms_by_name() -> None: + """Automatic, indexed, named, converted, and specified fields are exact.""" + + unsafe_sources = ( + """ +def f(df): + col = "person_support_channel" + return df.query("{} == 1".format(col)) +""", + """ +def f(df): + col = "person_support_channel" + return df.query("{0} == 1".format(col)) +""", + """ +def f(df): + col = "person_support_channel" + return df.query("{!s} == 1".format(col)) +""", + """ +def f(df): + col = "person_support_channel" + return df.query("{!r} == 1".format(col)) +""", + """ +def f(df): + col = "person_support_channel" + return df.query("{:s} == 1".format(col)) +""", + """ +def f(df): + return df.query( + "{col} == 1".format(col="person_support_channel") + ) +""", + """ +def f(df): + return df[ + "{entity}_support_channel".format(entity="person") + ] +""", + ) + + for source in unsafe_sources: + accesses = _source_spine_accesses(source) + assert accesses, source + assert any("person_support_channel" in item for item in accesses) + assert all("fail-closed" not in item for item in accesses) + + +def test_format_fields_preserve_benign_precision_and_opaque_failures() -> None: + """Fully static benign fields pass; unresolved fields fail closed.""" + + benign_sources = ( + """ +def f(df): + return df.query("{col} >= 18".format(col="age")) +""", + """ +def f(df): + return df.query("{0!s:>3} >= 18".format("age")) +""", + """ +def f(df): + return df.query("age * 2 >= 18") +""", + """ +def f(df): + return df.query("{{age}} == {{age}}".format()) +""", + ) + opaque_sources = ( + """ +def f(df, col): + return df.query("{col} == 1".format(col=col)) +""", + """ +def f(df): + return df.eval("{missing} == 1".format()) +""", + """ +def f(df, values): + return df.query("{} == 1".format(*values)) +""", + """ +def f(df, values): + return df.query("{col} == 1".format(**values)) +""", + ) + + for source in benign_sources: + assert _source_spine_accesses(source) == (), source + for source in opaque_sources: + accesses = _source_spine_accesses(source) + assert accesses, source + assert any("fail-closed" in item for item in accesses) + + def test_source_spine_ast_guard_covers_every_entity_grain() -> None: """Every US and benefit-unit grain's source identity is prohibited.""" From 9bf61b0ecbc3abdb89b7f5742b63ca9059e5e553 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 00:13:34 -0400 Subject: [PATCH 40/94] Apply strict pandas checks through method aliases --- PROGRESS.md | 18 +- .../tests/test_us_spine_blindness.py | 453 +++++++++++++++--- 2 files changed, 390 insertions(+), 81 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 0f32bfad..751def73 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -7,7 +7,9 @@ clean starting commit `19a7a1d`. Subscript resolution now catches all six round-4 evasions; loop and comprehension targets now propagate every static string choice or an opaque shadow; and the 54-module operator graph remains clean. `str.format` now resolves all requested static field forms precisely. -Method aliases and late-bound closure values remain in progress. +Direct and aliased strict pandas methods now share the same checks, and dynamic +DataFrame `getattr` is fail-closed. Late-bound closure values remain in +progress. No push or external mutation is authorized. @@ -43,12 +45,20 @@ No push or external mutation is authorized. controls for named benign formatting, conversions/specs, escaped braces, and multiplication syntax. The guard file now passes 19 tests with the graph clean. +- Tracked scoped aliases of `query`, `eval`, `filter`, and `get`, including + alias chains, explicit rebinding shadows, and static `getattr` aliases. +- Routed direct and aliased calls through shared strict handlers, including + hidden/expanded argument and opaque-key failures. +- Made dynamic `getattr` fail closed for inferred DataFrame/Frame containers + while distinguishing generic object and Series index access. +- Added direct-vs-alias exact-name tests for all four methods, benign and opaque + batteries, rebinding/parameter-shadow controls, and static/dynamic `getattr` + probes. The guard file now passes 22 tests with the 54-module graph clean. ## Next -- Commit complete format-field resolution. -- Add committed self-tests and implementation commits for method aliasing and - closure late binding. +- Commit strict-method aliasing and dynamic `getattr`. +- Add the closure late-binding assignment pre-pass and committed self-tests. - Add the cross-round completeness invariant and benign battery. - Run the guard file, the full `populace-build` suite, Ruff, and the 54-module graph scan. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index f9e2a54f..643aea87 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -66,6 +66,7 @@ "support_channel_column", } ) +_STRICT_COLUMN_METHODS = frozenset({"query", "eval", "filter", "get"}) _PUF_CLONE_OPERATOR_MODULES = ( "puf_capital_gains_tail.py", "puf_qrf_chain.py", @@ -636,6 +637,8 @@ def __init__(self, factory_aliases: set[str]) -> None: self.bindings: list[dict[str, str | None]] = [{}] self.constants: list[dict[str, object]] = [{}] self.column_containers: list[dict[str, bool]] = [{}] + self.attribute_containers: list[dict[str, bool]] = [{}] + self.method_aliases: list[dict[str, tuple[str, bool] | None]] = [{}] self.accesses: set[tuple[int, int, str]] = set() def _expression(self, node: ast.AST) -> str | None: @@ -662,6 +665,47 @@ def _column_container(self, node: ast.AST) -> bool: return self._column_container(node.func.value) return False + def _attribute_container(self, node: ast.AST) -> bool: + if isinstance(node, ast.Name): + for scope in reversed(self.attribute_containers): + if node.id in scope: + return scope[node.id] + return False + if isinstance(node, ast.Call): + if _call_name(node) in {"DataFrame", "table"}: + return True + if isinstance(node.func, ast.Attribute) and node.func.attr == "copy": + return self._attribute_container(node.func.value) + return False + + def _method_alias( + self, + node: ast.AST, + ) -> tuple[str, bool] | None: + if isinstance(node, ast.Name): + for scope in reversed(self.method_aliases): + if node.id in scope: + return scope[node.id] + return None + if isinstance(node, ast.Attribute) and node.attr in _STRICT_COLUMN_METHODS: + strict_opacity = node.attr != "get" or self._attribute_container(node.value) + return node.attr, strict_opacity + if ( + isinstance(node, ast.Call) + and _call_name(node) == "getattr" + and len(node.args) >= 2 + ): + attribute = _static_string_shape(node.args[1], self.constants) + if ( + attribute in _STRICT_COLUMN_METHODS + and _OPAQUE_STRING_PART not in attribute + ): + strict_opacity = attribute != "get" or self._attribute_container( + node.args[0] + ) + return attribute, strict_opacity + return None + def _bind(self, targets: list[ast.AST], value: ast.AST) -> None: description = self._expression(value) constant: object = _static_string_shape(value, self.constants) @@ -670,6 +714,8 @@ def _bind(self, targets: list[ast.AST], value: ast.AST) -> None: if constant is None: constant = _static_integer(value, self.constants) container = self._column_container(value) + attribute_container = self._attribute_container(value) + method_alias = self._method_alias(value) for target in targets: for name in _assigned_names(target): self.bindings[-1][name] = description @@ -678,6 +724,8 @@ def _bind(self, targets: list[ast.AST], value: ast.AST) -> None: # parameters and conditional reassignments — sol round 3). self.constants[-1][name] = constant self.column_containers[-1][name] = container + self.attribute_containers[-1][name] = attribute_container + self.method_aliases[-1][name] = method_alias def visit_Assign(self, node: ast.Assign) -> None: self.visit(node.value) @@ -700,6 +748,8 @@ def visit_AnnAssign(self, node: ast.AnnAssign) -> None: self.bindings[-1][name] = None self.constants[-1][name] = None self.column_containers[-1][name] = False + self.attribute_containers[-1][name] = False + self.method_aliases[-1][name] = None return self.visit(node.value) if isinstance(node.target, ast.Subscript): @@ -719,6 +769,8 @@ def _bind_iteration_target( self.bindings[-1][name] = None self.constants[-1][name] = values self.column_containers[-1][name] = False + self.attribute_containers[-1][name] = False + self.method_aliases[-1][name] = None def visit_For(self, node: ast.For) -> None: self.visit(node.iter) @@ -739,6 +791,8 @@ def _visit_comprehension( self.bindings.append({}) self.constants.append({}) self.column_containers.append({}) + self.attribute_containers.append({}) + self.method_aliases.append({}) for generator in node.generators: self.visit(generator.iter) self._bind_iteration_target( @@ -752,6 +806,8 @@ def _visit_comprehension( self.visit(node.value) else: self.visit(node.elt) + self.method_aliases.pop() + self.attribute_containers.pop() self.column_containers.pop() self.constants.pop() self.bindings.pop() @@ -799,15 +855,34 @@ def _visit_function( and argument.arg not in {"self", "cls"} for argument in arguments } + attribute_containers = { + argument.arg: ( + argument.annotation is None and argument.arg not in {"self", "cls"} + ) + or ( + argument.annotation is not None + and any( + marker in ast.unparse(argument.annotation) + for marker in ("DataFrame", "Frame") + ) + ) + for argument in arguments + } if node.args.vararg is not None: containers[node.args.vararg.arg] = False + attribute_containers[node.args.vararg.arg] = False if node.args.kwarg is not None: containers[node.args.kwarg.arg] = False + attribute_containers[node.args.kwarg.arg] = False self.bindings.append(local) self.constants.append({name: None for name in local}) self.column_containers.append(containers) + self.attribute_containers.append(attribute_containers) + self.method_aliases.append({name: None for name in local}) for statement in node.body: self.visit(statement) + self.method_aliases.pop() + self.attribute_containers.pop() self.column_containers.pop() self.constants.pop() self.bindings.pop() @@ -827,6 +902,10 @@ def visit_Lambda(self, node: ast.Lambda) -> None: *node.args.kwonlyargs, ) } + if node.args.vararg is not None: + local[node.args.vararg.arg] = None + if node.args.kwarg is not None: + local[node.args.kwarg.arg] = None containers = { argument.arg: argument.annotation is None and argument.arg not in {"self", "cls"} @@ -836,10 +915,21 @@ def visit_Lambda(self, node: ast.Lambda) -> None: *node.args.kwonlyargs, ) } + attribute_containers = dict(containers) + if node.args.vararg is not None: + containers[node.args.vararg.arg] = False + attribute_containers[node.args.vararg.arg] = False + if node.args.kwarg is not None: + containers[node.args.kwarg.arg] = False + attribute_containers[node.args.kwarg.arg] = False self.bindings.append(local) self.constants.append({name: None for name in local}) self.column_containers.append(containers) + self.attribute_containers.append(attribute_containers) + self.method_aliases.append({name: None for name in local}) self.visit(node.body) + self.method_aliases.pop() + self.attribute_containers.pop() self.column_containers.pop() self.constants.pop() self.bindings.pop() @@ -862,6 +952,148 @@ def visit_Lambda(self, node: ast.Lambda) -> None: def _poison(self, name: str) -> None: self.constants[-1][name] = None + def _visit_getattr(self, node: ast.Call) -> None: + if len(node.args) < 2: + return + attribute = _static_string_shape(node.args[1], self.constants) + if attribute is None or _OPAQUE_STRING_PART in attribute: + if self._attribute_container(node.args[0]): + self._record( + node, + "getattr with an unresolvable dynamic attribute (fail-closed)", + ) + elif _is_source_column_shape(attribute): + self._record( + node, + f"getattr using source column {attribute!r}", + ) + + def _visit_get_call( + self, + node: ast.Call, + *, + strict_opacity: bool, + ) -> None: + key = _call_argument(node, position=0, keyword="key") + if key is None: + if strict_opacity and (node.args or node.keywords): + self._record( + node, + ".get() with hidden or expanded arguments (fail-closed)", + ) + return + resolved = _static_string_values(key, self.constants) + column = _subscript_source_expression( + key, + bindings=self.bindings, + factory_aliases=self.factory_aliases, + ) + if resolved is None: + if column is not None: + self._record(node, f".get() using {column}") + elif strict_opacity: + self._record( + node, + ".get() with an unresolvable dynamic key (fail-closed)", + ) + return + if any(_OPAQUE_STRING_PART in item for item in resolved): + if strict_opacity or column is not None: + self._record( + node, + ".get() with an unresolvable dynamic key (fail-closed)", + ) + return + for item in resolved: + if _is_source_column_shape(item): + self._record( + node, + f".get() using source column {item!r}", + ) + + def _visit_query_or_eval_call( + self, + node: ast.Call, + *, + method: str, + ) -> None: + expression = _call_argument(node, position=0, keyword="expr") + if expression is None and (node.args or node.keywords): + self._record( + node, + f".{method}() with hidden or expanded arguments (fail-closed)", + ) + return + if expression is None: + return + shape = _static_string_shape(expression, self.constants) + if shape is None or _OPAQUE_STRING_PART in shape: + self._record( + node, + f".{method}() with an unresolvable dynamic expression (fail-closed)", + ) + return + for token in re.findall(r"[\w*?\[\]-]+", shape): + if _is_source_column_shape(token): + self._record( + node, + f".{method}() using source column {token!r}", + ) + break + + def _visit_filter_call(self, node: ast.Call) -> None: + items = _call_argument(node, position=0, keyword="items") + if items is None and (node.args or node.keywords): + self._record( + node, + ".filter() with hidden or expanded arguments (fail-closed)", + ) + return + if items is None: + return + resolved = _static_string_list(items, self.constants) + if resolved is None: + column = _subscript_source_expression( + items, + bindings=self.bindings, + factory_aliases=self.factory_aliases, + ) + if column is not None: + self._record(node, f".filter(items=...) using {column}") + else: + self._record( + node, + ".filter(items=...) with an unresolvable dynamic " + "list (fail-closed)", + ) + return + if any(_OPAQUE_STRING_PART in item for item in resolved): + self._record( + node, + ".filter(items=...) with an unresolvable dynamic list (fail-closed)", + ) + return + for item in resolved: + if _is_source_column_shape(item): + self._record( + node, + f".filter(items=...) using source column {item!r}", + ) + + def _visit_strict_method_call( + self, + node: ast.Call, + *, + method: str, + strict_opacity: bool, + ) -> None: + if method == "get": + self._visit_get_call(node, strict_opacity=strict_opacity) + elif method in {"query", "eval"}: + self._visit_query_or_eval_call(node, method=method) + else: + self._visit_filter_call(node) + def visit_Call(self, node: ast.Call) -> None: if ( isinstance(node.func, ast.Attribute) @@ -872,83 +1104,15 @@ def visit_Call(self, node: ast.Call) -> None: name = _call_name(node) if name in self.factory_aliases: self._record(node, f"call to {name}") - elif name == "getattr" and len(node.args) >= 2: - attribute = self._expression(node.args[1]) - if attribute is not None: - self._record(node, f"getattr using {attribute}") - elif isinstance(node.func, ast.Attribute) and name == "get": - key = _call_argument(node, position=0, keyword="key") - if key is not None: - column = _subscript_source_expression( - key, - bindings=self.bindings, - factory_aliases=self.factory_aliases, - ) - if column is not None: - self._record(node, f".get() using {column}") - elif isinstance(node.func, ast.Attribute) and name in {"query", "eval"}: - expression = _call_argument(node, position=0, keyword="expr") - if expression is None and (node.args or node.keywords): - self._record( - node, - f".{name}() with hidden or expanded arguments (fail-closed)", - ) - elif expression is not None: - shape = _static_string_shape(expression, self.constants) - if shape is None or _OPAQUE_STRING_PART in shape: - # Fail closed: an expression the guard cannot resolve - # statically could name a guarded column — opacity is - # a violation, not a pass (sol #583 round 2). - self._record( - node, - f".{name}() with an unresolvable dynamic expression " - "(fail-closed)", - ) - else: - for token in re.findall(r"[\w*?\[\]-]+", shape): - if _is_source_column_shape(token): - self._record( - node, - f".{name}() using source column {token!r}", - ) - break - elif isinstance(node.func, ast.Attribute) and name == "filter": - items = _call_argument(node, position=0, keyword="items") - if items is None and (node.args or node.keywords): - self._record( - node, - ".filter() with hidden or expanded arguments (fail-closed)", - ) - elif items is not None: - resolved = _static_string_list(items, self.constants) - if resolved is None: - column = _subscript_source_expression( - items, - bindings=self.bindings, - factory_aliases=self.factory_aliases, - ) - if column is not None: - self._record(node, f".filter(items=...) using {column}") - else: - self._record( - node, - ".filter(items=...) with an unresolvable dynamic " - "list (fail-closed)", - ) - elif any(_OPAQUE_STRING_PART in item for item in resolved): - self._record( - node, - ".filter(items=...) with an unresolvable dynamic " - "list (fail-closed)", - ) - else: - for item in resolved: - if _is_source_column_shape(item): - self._record( - node, - f".filter(items=...) using source column {item!r}", - ) - break + elif name == "getattr": + self._visit_getattr(node) + elif (method_alias := self._method_alias(node.func)) is not None: + method, strict_opacity = method_alias + self._visit_strict_method_call( + node, + method=method, + strict_opacity=strict_opacity, + ) self.generic_visit(node) def visit_Attribute(self, node: ast.Attribute) -> None: @@ -1747,6 +1911,141 @@ def f(df, values): assert any("fail-closed" in item for item in accesses) +def test_strict_method_aliases_match_direct_column_checks() -> None: + """Aliases of query, eval, filter, and get retain exact strict behavior.""" + + unsafe = { + "query": '"person_support_channel == 1"', + "eval": '"person_support_channel == 1"', + "filter": 'items=["person_support_channel"]', + "get": '"person_support_channel"', + } + benign = { + "query": '"age >= 18"', + "eval": '"age + 1"', + "filter": 'items=["age", "income"]', + "get": '"age"', + } + + for method, arguments in unsafe.items(): + direct = f""" +def f(df): + return df.{method}({arguments}) +""" + aliased = f""" +def f(df): + method = df.{method} + return method({arguments}) +""" + direct_accesses = _source_spine_accesses(direct) + alias_accesses = _source_spine_accesses(aliased) + assert direct_accesses, direct + assert alias_accesses, aliased + assert any("person_support_channel" in item for item in alias_accesses) + assert all("fail-closed" not in item for item in alias_accesses) + + for method, arguments in benign.items(): + source = f""" +def f(df): + method = df.{method} + return method({arguments}) +""" + assert _source_spine_accesses(source) == (), source + + +def test_strict_method_aliases_fail_closed_and_shadow_precisely() -> None: + """Opaque alias arguments fail; rebinding and parameters stop stale aliases.""" + + opaque_sources = ( + """ +def f(df, expr): + query = df.query + return query(expr) +""", + """ +def f(df, expr): + evaluate = df.eval + return evaluate(expr) +""", + """ +def f(df, columns): + select = df.filter + return select(items=columns) +""", + """ +def f(df, key): + get = df.get + return get(key) +""", + """ +def f(df, kwargs): + query = df.query + return query(**kwargs) +""", + ) + shadowed_sources = ( + """ +def f(df): + query = df.query + query = print + return query("age >= 18") +""", + """ +query = df.query +def f(query): + return query("age >= 18") +""", + ) + + for source in opaque_sources: + accesses = _source_spine_accesses(source) + assert accesses, source + assert any("fail-closed" in item for item in accesses) + for source in shadowed_sources: + assert _source_spine_accesses(source) == (), source + + +def test_getattr_column_access_resolves_or_fails_closed() -> None: + """Static attributes are checked by name; dynamic table attrs are opaque.""" + + static_alias = """ +def f(df): + query = getattr(df, "query") + return query("person_support_channel == 1") +""" + immediate_alias = """ +def f(df): + return getattr(df, "eval")("person_support_channel == 1") +""" + guarded_attribute = """ +def f(df): + return getattr(df, "person_support_channel") +""" + dynamic_attribute = """ +def f(df, attribute): + return getattr(df, attribute) +""" + benign_attribute = """ +def f(df): + attribute = "age" + return getattr(df, attribute) +""" + generic_object = """ +def f(obj: object, attribute: str): + return getattr(obj, attribute) +""" + + for source in (static_alias, immediate_alias, guarded_attribute): + accesses = _source_spine_accesses(source) + assert accesses, source + assert any("person_support_channel" in item for item in accesses) + dynamic_accesses = _source_spine_accesses(dynamic_attribute) + assert dynamic_accesses + assert any("fail-closed" in item for item in dynamic_accesses) + assert _source_spine_accesses(benign_attribute) == () + assert _source_spine_accesses(generic_object) == () + + def test_source_spine_ast_guard_covers_every_entity_grain() -> None: """Every US and benefit-unit grain's source identity is prohibited.""" From 91b41ea3934e78c74fba0421af5843163936f853 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 00:17:48 -0400 Subject: [PATCH 41/94] Fail closed on late-bound closure values --- PROGRESS.md | 20 +- .../tests/test_us_spine_blindness.py | 340 ++++++++++++++++-- 2 files changed, 317 insertions(+), 43 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 751def73..94562e31 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -8,8 +8,9 @@ round-4 evasions; loop and comprehension targets now propagate every static string choice or an opaque shadow; and the 54-module operator graph remains clean. `str.format` now resolves all requested static field forms precisely. Direct and aliased strict pandas methods now share the same checks, and dynamic -DataFrame `getattr` is fail-closed. Late-bound closure values remain in -progress. +DataFrame `getattr` is fail-closed. Closure free variables now obey lexical +late-binding assignment counts. The five requested subsystems are implemented; +the cross-round invariant, final docstrings, and full validation remain. No push or external mutation is authorized. @@ -54,11 +55,22 @@ No push or external mutation is authorized. - Added direct-vs-alias exact-name tests for all four methods, benign and opaque batteries, rebinding/parameter-shadow controls, and static/dynamic `getattr` probes. The guard file now passes 22 tests with the 54-module graph clean. +- Added a lexical scope pre-pass that counts binding sites without descending + nested function/class bodies and pre-shadows all Python locals. +- Deferred nested function-body analysis until the enclosing scope's bindings + are complete; free names with exactly one defining-scope assignment resolve, + while multi-assignment free names become explicit opacity. +- Preserved the same late-binding rule for lambdas and strict method aliases, + so a rebound alias cannot degrade to an ignored ordinary Name call. +- Added the exact reviewer late-bound closure, module-level and lambda variants, + stable guarded/benign controls, later-local shadowing, and stable/rebound + alias tests. The guard file now passes 24 tests with the graph clean. ## Next -- Commit strict-method aliasing and dynamic `getattr`. -- Add the closure late-binding assignment pre-pass and committed self-tests. +- Commit closure late-binding handling. +- Add the parametrized cross-round completeness invariant, expanded benign + battery, exact 54-module count assertion, and completed-contract docstrings. - Add the cross-round completeness invariant and benign battery. - Run the guard file, the full `populace-build` suite, Ruff, and the 54-module graph scan. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 643aea87..e1e1c17c 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -428,6 +428,7 @@ def _static_string_shape( _OPAQUE_STATIC_VALUE = object() _OPAQUE_STRING_PART = "\N{OBJECT REPLACEMENT CHARACTER}" +_OPAQUE_METHOD_ALIAS = ("", True) def _static_format_value( @@ -631,6 +632,90 @@ def _assigned_names(target: ast.AST) -> tuple[str, ...]: return () +class _ScopeAssignmentCounter(ast.NodeVisitor): + """Count binding sites in one lexical scope, excluding nested scopes.""" + + def __init__(self) -> None: + self.counts: dict[str, int] = {} + + def _count(self, name: str) -> None: + self.counts[name] = self.counts.get(name, 0) + 1 + + def visit_Name(self, node: ast.Name) -> None: + if isinstance(node.ctx, (ast.Store, ast.Del)): + self._count(node.id) + + def _visit_defaults_and_decorators( + self, + node: ast.FunctionDef | ast.AsyncFunctionDef, + ) -> None: + self._count(node.name) + for decorator in node.decorator_list: + self.visit(decorator) + for default in (*node.args.defaults, *node.args.kw_defaults): + if default is not None: + self.visit(default) + + def visit_FunctionDef(self, node: ast.FunctionDef) -> None: + self._visit_defaults_and_decorators(node) + + def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None: + self._visit_defaults_and_decorators(node) + + def visit_Lambda(self, node: ast.Lambda) -> None: + for default in (*node.args.defaults, *node.args.kw_defaults): + if default is not None: + self.visit(default) + + def visit_ClassDef(self, node: ast.ClassDef) -> None: + self._count(node.name) + for expression in (*node.decorator_list, *node.bases): + self.visit(expression) + for keyword in node.keywords: + self.visit(keyword.value) + + def visit_Import(self, node: ast.Import) -> None: + for imported in node.names: + self._count(imported.asname or imported.name.split(".", maxsplit=1)[0]) + + def visit_ImportFrom(self, node: ast.ImportFrom) -> None: + for imported in node.names: + self._count(imported.asname or imported.name) + + def visit_ExceptHandler(self, node: ast.ExceptHandler) -> None: + if node.name is not None: + self._count(node.name) + if node.type is not None: + self.visit(node.type) + for statement in node.body: + self.visit(statement) + + def visit_ListComp(self, node: ast.ListComp) -> None: + self.visit(node.generators[0].iter) + + def visit_SetComp(self, node: ast.SetComp) -> None: + self.visit(node.generators[0].iter) + + def visit_DictComp(self, node: ast.DictComp) -> None: + self.visit(node.generators[0].iter) + + def visit_GeneratorExp(self, node: ast.GeneratorExp) -> None: + self.visit(node.generators[0].iter) + + +def _scope_assignment_counts( + body: list[ast.stmt], + *, + parameters: tuple[str, ...] = (), +) -> dict[str, int]: + counter = _ScopeAssignmentCounter() + for statement in body: + counter.visit(statement) + for name in parameters: + counter._count(name) + return counter.counts + + class _SourceReadVisitor(ast.NodeVisitor): def __init__(self, factory_aliases: set[str]) -> None: self.factory_aliases = factory_aliases @@ -639,6 +724,8 @@ def __init__(self, factory_aliases: set[str]) -> None: self.column_containers: list[dict[str, bool]] = [{}] self.attribute_containers: list[dict[str, bool]] = [{}] self.method_aliases: list[dict[str, tuple[str, bool] | None]] = [{}] + self.method_alias_history: list[set[str]] = [set()] + self.assignment_counts: list[dict[str, int]] = [{}] self.accesses: set[tuple[int, int, str]] = set() def _expression(self, node: ast.AST) -> str | None: @@ -726,6 +813,8 @@ def _bind(self, targets: list[ast.AST], value: ast.AST) -> None: self.column_containers[-1][name] = container self.attribute_containers[-1][name] = attribute_container self.method_aliases[-1][name] = method_alias + if method_alias is not None: + self.method_alias_history[-1].add(name) def visit_Assign(self, node: ast.Assign) -> None: self.visit(node.value) @@ -793,6 +882,8 @@ def _visit_comprehension( self.column_containers.append({}) self.attribute_containers.append({}) self.method_aliases.append({}) + self.method_alias_history.append(set()) + self.assignment_counts.append({}) for generator in node.generators: self.visit(generator.iter) self._bind_iteration_target( @@ -806,6 +897,8 @@ def _visit_comprehension( self.visit(node.value) else: self.visit(node.elt) + self.assignment_counts.pop() + self.method_alias_history.pop() self.method_aliases.pop() self.attribute_containers.pop() self.column_containers.pop() @@ -824,6 +917,42 @@ def visit_DictComp(self, node: ast.DictComp) -> None: def visit_GeneratorExp(self, node: ast.GeneratorExp) -> None: self._visit_comprehension(node) + def _visit_scope_statements(self, body: list[ast.stmt]) -> None: + deferred: list[ast.FunctionDef | ast.AsyncFunctionDef] = [] + for statement in body: + if isinstance(statement, (ast.FunctionDef, ast.AsyncFunctionDef)): + deferred.append(statement) + else: + self.visit(statement) + for function in deferred: + self.visit(function) + + def visit_Module(self, node: ast.Module) -> None: + counts = _scope_assignment_counts(node.body) + self.assignment_counts[0] = counts + for name in counts: + self.bindings[0].setdefault(name, None) + self.constants[0].setdefault(name, None) + self.column_containers[0].setdefault(name, False) + self.attribute_containers[0].setdefault(name, False) + self.method_aliases[0].setdefault(name, None) + self._visit_scope_statements(node.body) + + def _unstable_outer_names(self) -> dict[str, bool]: + unstable: dict[str, bool] = {} + seen: set[str] = set() + for index in range(len(self.constants) - 1, -1, -1): + names = ( + set(self.assignment_counts[index]) + | set(self.constants[index]) + | self.method_alias_history[index] + ) + for name in names - seen: + seen.add(name) + if self.assignment_counts[index].get(name, 0) != 1: + unstable[name] = name in self.method_alias_history[index] + return unstable + def _visit_function( self, node: ast.FunctionDef | ast.AsyncFunctionDef, @@ -833,23 +962,23 @@ def _visit_function( for default in (*node.args.defaults, *node.args.kw_defaults): if default is not None: self.visit(default) - local: dict[str, str | None] = { - argument.arg: None - for argument in ( - *node.args.posonlyargs, - *node.args.args, - *node.args.kwonlyargs, - ) - } - if node.args.vararg is not None: - local[node.args.vararg.arg] = None - if node.args.kwarg is not None: - local[node.args.kwarg.arg] = None arguments = ( *node.args.posonlyargs, *node.args.args, *node.args.kwonlyargs, ) + parameter_names = tuple(argument.arg for argument in arguments) + if node.args.vararg is not None: + parameter_names += (node.args.vararg.arg,) + if node.args.kwarg is not None: + parameter_names += (node.args.kwarg.arg,) + counts = _scope_assignment_counts( + node.body, + parameters=parameter_names, + ) + unstable_outer = self._unstable_outer_names() + local_names = set(counts) | set(unstable_outer) + local: dict[str, str | None] = dict.fromkeys(local_names) containers = { argument.arg: argument.annotation is None and argument.arg not in {"self", "cls"} @@ -874,13 +1003,32 @@ def _visit_function( if node.args.kwarg is not None: containers[node.args.kwarg.arg] = False attribute_containers[node.args.kwarg.arg] = False + for name in local_names: + containers.setdefault(name, False) + attribute_containers.setdefault(name, False) + method_aliases = { + name: ( + _OPAQUE_METHOD_ALIAS + if name not in counts and unstable_outer[name] + else None + ) + for name in local_names + } + alias_history = { + name + for name, value in method_aliases.items() + if value == _OPAQUE_METHOD_ALIAS + } self.bindings.append(local) - self.constants.append({name: None for name in local}) + self.constants.append(dict.fromkeys(local_names)) self.column_containers.append(containers) self.attribute_containers.append(attribute_containers) - self.method_aliases.append({name: None for name in local}) - for statement in node.body: - self.visit(statement) + self.method_aliases.append(method_aliases) + self.method_alias_history.append(alias_history) + self.assignment_counts.append(counts) + self._visit_scope_statements(node.body) + self.assignment_counts.pop() + self.method_alias_history.pop() self.method_aliases.pop() self.attribute_containers.pop() self.column_containers.pop() @@ -894,26 +1042,28 @@ def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None: self._visit_function(node) def visit_Lambda(self, node: ast.Lambda) -> None: - local = { - argument.arg: None - for argument in ( - *node.args.posonlyargs, - *node.args.args, - *node.args.kwonlyargs, - ) - } + arguments = ( + *node.args.posonlyargs, + *node.args.args, + *node.args.kwonlyargs, + ) + parameter_names = tuple(argument.arg for argument in arguments) if node.args.vararg is not None: - local[node.args.vararg.arg] = None + parameter_names += (node.args.vararg.arg,) if node.args.kwarg is not None: - local[node.args.kwarg.arg] = None + parameter_names += (node.args.kwarg.arg,) + counter = _ScopeAssignmentCounter() + counter.visit(node.body) + for name in parameter_names: + counter._count(name) + counts = counter.counts + unstable_outer = self._unstable_outer_names() + local_names = set(counts) | set(unstable_outer) + local: dict[str, str | None] = dict.fromkeys(local_names) containers = { argument.arg: argument.annotation is None and argument.arg not in {"self", "cls"} - for argument in ( - *node.args.posonlyargs, - *node.args.args, - *node.args.kwonlyargs, - ) + for argument in arguments } attribute_containers = dict(containers) if node.args.vararg is not None: @@ -922,12 +1072,32 @@ def visit_Lambda(self, node: ast.Lambda) -> None: if node.args.kwarg is not None: containers[node.args.kwarg.arg] = False attribute_containers[node.args.kwarg.arg] = False + for name in local_names: + containers.setdefault(name, False) + attribute_containers.setdefault(name, False) + method_aliases = { + name: ( + _OPAQUE_METHOD_ALIAS + if name not in counts and unstable_outer[name] + else None + ) + for name in local_names + } + alias_history = { + name + for name, value in method_aliases.items() + if value == _OPAQUE_METHOD_ALIAS + } self.bindings.append(local) - self.constants.append({name: None for name in local}) + self.constants.append(dict.fromkeys(local_names)) self.column_containers.append(containers) self.attribute_containers.append(attribute_containers) - self.method_aliases.append({name: None for name in local}) + self.method_aliases.append(method_aliases) + self.method_alias_history.append(alias_history) + self.assignment_counts.append(counts) self.visit(node.body) + self.assignment_counts.pop() + self.method_alias_history.pop() self.method_aliases.pop() self.attribute_containers.pop() self.column_containers.pop() @@ -1107,12 +1277,18 @@ def visit_Call(self, node: ast.Call) -> None: elif name == "getattr": self._visit_getattr(node) elif (method_alias := self._method_alias(node.func)) is not None: - method, strict_opacity = method_alias - self._visit_strict_method_call( - node, - method=method, - strict_opacity=strict_opacity, - ) + if method_alias == _OPAQUE_METHOD_ALIAS: + self._record( + node, + "call through an opaque late-bound method alias (fail-closed)", + ) + else: + method, strict_opacity = method_alias + self._visit_strict_method_call( + node, + method=method, + strict_opacity=strict_opacity, + ) self.generic_visit(node) def visit_Attribute(self, node: ast.Attribute) -> None: @@ -2046,6 +2222,92 @@ def f(obj: object, attribute: str): assert _source_spine_accesses(generic_object) == () +def test_closure_free_names_obey_late_binding_assignment_counts() -> None: + """Multi-assignment free names are opaque; stable names remain exact.""" + + late_bound = """ +def outer(df): + expr = "age >= 18" + def inner(): + return df.query(expr) + expr = "person_support_channel == 1" + return inner() +""" + module_late_bound = """ +expr = "age >= 18" +def f(df): + return df.query(expr) +expr = "person_support_channel == 1" +""" + stable_guarded = """ +def outer(df): + expr = "person_support_channel == 1" + def inner(): + return df.query(expr) + return inner() +""" + stable_benign = """ +def outer(df): + expr = "age >= 18" + def inner(): + return df.query(expr) + return inner() +""" + later_local_shadow = """ +expr = "person_support_channel == 1" +def outer(df): + def inner(): + return df.query(expr) + expr = "age >= 18" + return inner() +""" + lambda_late_bound = """ +def outer(df): + expr = "age >= 18" + inner = lambda: df.query(expr) + expr = "person_support_channel == 1" + return inner() +""" + + for source in (late_bound, module_late_bound, lambda_late_bound): + accesses = _source_spine_accesses(source) + assert accesses, source + assert any("fail-closed" in item for item in accesses) + guarded_accesses = _source_spine_accesses(stable_guarded) + assert guarded_accesses + assert any("person_support_channel" in item for item in guarded_accesses) + assert all("fail-closed" not in item for item in guarded_accesses) + assert _source_spine_accesses(stable_benign) == () + assert _source_spine_accesses(later_local_shadow) == () + + +def test_closure_method_aliases_are_stable_or_explicitly_opaque() -> None: + """Late alias rebinding cannot turn a strict call into a silent Name call.""" + + stable_alias = """ +def outer(df): + query = df.query + def inner(): + return query("person_support_channel == 1") + return inner() +""" + rebound_alias = """ +def outer(df): + query = df.query + def inner(): + return query("person_support_channel == 1") + query = print + return inner() +""" + + stable_accesses = _source_spine_accesses(stable_alias) + assert stable_accesses + assert any("person_support_channel" in item for item in stable_accesses) + rebound_accesses = _source_spine_accesses(rebound_alias) + assert rebound_accesses + assert any("fail-closed" in item for item in rebound_accesses) + + def test_source_spine_ast_guard_covers_every_entity_grain() -> None: """Every US and benefit-unit grain's source identity is prohibited.""" From ecea64e13d275d0d6cddda4fddd1d58eea441534 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 00:26:59 -0400 Subject: [PATCH 42/94] Pin the complete spine-blindness guard contract --- PROGRESS.md | 24 +- .../tests/test_us_spine_blindness.py | 407 +++++++++++++++++- 2 files changed, 420 insertions(+), 11 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 94562e31..e6d65412 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -9,8 +9,10 @@ string choice or an opaque shadow; and the 54-module operator graph remains clean. `str.format` now resolves all requested static field forms precisely. Direct and aliased strict pandas methods now share the same checks, and dynamic DataFrame `getattr` is fail-closed. Closure free variables now obey lexical -late-binding assignment counts. The five requested subsystems are implemented; -the cross-round invariant, final docstrings, and full validation remain. +late-binding assignment counts. The five requested subsystems, consolidated +rounds 2-4 completeness invariant, benign battery, exact graph cardinality, +and completed-contract docstrings are implemented. Adversarial audit and full +validation remain. No push or external mutation is authorized. @@ -65,13 +67,23 @@ No push or external mutation is authorized. - Added the exact reviewer late-bound closure, module-level and lambda variants, stable guarded/benign controls, later-local shadowing, and stable/rebound alias tests. The guard file now passes 24 tests with the graph clean. +- Replaced the guard module and scan-test docstrings with the completed + resolve-and-check-or-fail-closed contract and an explicit surface inventory. +- Added one parametrized invariant covering bound names, parameter f-strings, + expanded kwargs, shadows, conditionals, mutation, all requested format + variants, all six round-4 subscript evasions, all four method aliases, + dynamic `getattr`, and late closure binding. +- Added a parametrized benign battery spanning every corresponding static + surface and typed non-column controls. +- Pinned the multispine tool graph to exactly 54 runtime modules. +- Replayed the expanded guard file (70 passed), focused Ruff, and + `git diff --check`; the exact 54-module graph remains clean. ## Next -- Commit closure late-binding handling. -- Add the parametrized cross-round completeness invariant, expanded benign - battery, exact 54-module count assertion, and completed-contract docstrings. -- Add the cross-round completeness invariant and benign battery. +- Commit the consolidated invariant, benign battery, graph pin, and docstrings. +- Resolve any acceptance-relevant findings from the independent adversarial + audit and replay the guard. - Run the guard file, the full `populace-build` suite, Ruff, and the 54-module graph scan. - Write `/private/tmp/583_fix2_handoff.md` with per-subsystem rules, binding diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index e1e1c17c..ff8632ea 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -1,4 +1,13 @@ -"""Structural guard for source-spine-blind US population operators.""" +"""Fail-closed guard for source-spine-blind US population operators. + +In every non-owner operator module, each column-access surface must either +resolve to static string choices that are checked by name or be recorded as a +fail-closed violation; there is no silent third state. Covered surfaces are +subscript and ``.loc`` reads/writes, attributes and dynamic ``getattr``, +canonical column factories, and direct or aliased pandas ``get``, ``filter``, +``query``, and ``eval`` calls. Loop/comprehension choices and closure free +variables obey the same contract. +""" from __future__ import annotations @@ -8,6 +17,8 @@ from pathlib import Path from string import Formatter +import pytest + _REPOSITORY_ROOT = Path(__file__).resolve().parents[3] _US_RUNTIME = ( Path(__file__).resolve().parents[1] / "src" / "populace" / "build" / "us_runtime" @@ -917,6 +928,9 @@ def visit_DictComp(self, node: ast.DictComp) -> None: def visit_GeneratorExp(self, node: ast.GeneratorExp) -> None: self._visit_comprehension(node) + def visit_TypeAlias(self, node: ast.TypeAlias) -> None: + """Type parameters are not executable column selectors.""" + def _visit_scope_statements(self, body: list[ast.stmt]) -> None: deferred: list[ast.FunctionDef | ast.AsyncFunctionDef] = [] for statement in body: @@ -1343,7 +1357,7 @@ def visit_Subscript(self, node: ast.Subscript) -> None: def _source_spine_accesses(source: str) -> tuple[str, ...]: - """Describe data reads that resolve any entity's source-spine identity.""" + """Describe named or fail-closed violations on every column surface.""" tree = ast.parse(source) aliases = _factory_aliases(tree) @@ -1502,12 +1516,13 @@ def test_us_runtime_frame_rebuilds_preserve_immutable_metadata() -> None: def test_runtime_population_operators_are_source_spine_blind() -> None: - """Only provenance owners may resolve any entity's source-spine identity. + """Every operator column surface resolves-and-checks or fails closed. The guard parses executable syntax rather than searching raw text, so comments, docstrings, and source-manifest declarations may explain the - invariant without creating an exception. Data access through a concrete - column, a dynamic string, ``getattr``, or a canonical factory fails. + invariant without creating an exception. It covers subscript/``.loc`` + reads and writes, attributes/``getattr``, canonical factories, and direct + or aliased ``get``, ``filter``, ``query``, and ``eval`` calls. """ missing_owners = sorted( @@ -1565,6 +1580,10 @@ def test_pool_build_tool_import_graph_is_source_spine_blind() -> None: for tool in _SPINE_BLIND_BUILD_TOOLS: runtime_graph, missing_modules = _us_runtime_import_graph(tool) + assert len(runtime_graph) == 54, ( + f"{tool.name} must reach the pinned 54-module runtime graph; " + f"reached {len(runtime_graph)}" + ) assert not missing_modules, ( f"{tool.name} imports unresolved US runtime modules: {missing_modules}" ) @@ -2308,6 +2327,384 @@ def inner(): assert any("fail-closed" in item for item in rebound_accesses) +@pytest.mark.parametrize( + ("evasion", "source"), + ( + ( + "round2-bound-query", + """ +def f(df): + expr = "person_support_channel == 1" + return df.query(expr) +""", + ), + ( + "round2-bound-eval", + """ +def f(df): + expr = "person_support_channel == 1" + return df.eval(expr) +""", + ), + ( + "round2-bound-filter", + """ +def f(df): + columns = ["person_support_channel"] + return df.filter(items=columns) +""", + ), + ( + "round2-bound-fstring", + """ +def f(df): + column = "person_support_channel" + return df.query(f"{column} == 1") +""", + ), + ( + "round3-parameter-fstring", + """ +def f(df, column): + return df.query(f"{column} == 1") +""", + ), + ( + "round3-expanded-kwargs", + """ +def f(df, kwargs): + return df.query(**kwargs) +""", + ), + ( + "round3-shadowed-parameter", + """ +column = "person_support_channel" +def f(df, column): + return df.query(f"{column} == 1") +""", + ), + ( + "round3-conditional", + """ +def f(df, flag): + column = "age" if flag else "person_support_channel" + return df.query(f"{column} == 1") +""", + ), + ( + "round3-mutated-list", + """ +def f(df): + columns = ["age"] + columns.append("person_support_channel") + return df.filter(items=columns) +""", + ), + ( + "format-automatic", + """ +def f(df): + return df.query("{} == 1".format("person_support_channel")) +""", + ), + ( + "format-indexed", + """ +def f(df): + return df.query("{0} == 1".format("person_support_channel")) +""", + ), + ( + "format-named", + """ +def f(df): + return df.query( + "{column} == 1".format(column="person_support_channel") + ) +""", + ), + ( + "format-convert-s", + """ +def f(df): + return df.query("{!s} == 1".format("person_support_channel")) +""", + ), + ( + "format-convert-r", + """ +def f(df): + return df.query("{!r} == 1".format("person_support_channel")) +""", + ), + ( + "format-spec", + """ +def f(df): + return df.query("{:s} == 1".format("person_support_channel")) +""", + ), + ( + "subscript-walrus", + """ +def f(df): + return df[(column := "person_support_channel")] +""", + ), + ( + "subscript-nested-call", + """ +def f(df): + def column(): + return "person_support_channel" + return df[column()] +""", + ), + ( + "subscript-dict-indirection", + """ +COLUMNS = {"source": "person_support_channel"} +def f(df): + return df[COLUMNS["source"]] +""", + ), + ( + "subscript-multiplication", + """ +def f(df): + return df["person_support_channel" * 1] +""", + ), + ( + "subscript-percent-format", + """ +def f(df): + return df["%s_support_channel" % "person"] +""", + ), + ( + "subscript-replace-chain", + """ +def f(df): + return df["person_x".replace("x", "support_channel")] +""", + ), + ( + "aliased-query", + """ +def f(df): + query = df.query + return query("person_support_channel == 1") +""", + ), + ( + "aliased-eval", + """ +def f(df): + evaluate = df.eval + return evaluate("person_support_channel == 1") +""", + ), + ( + "aliased-filter", + """ +def f(df): + select = df.filter + return select(items=["person_support_channel"]) +""", + ), + ( + "aliased-get", + """ +def f(df): + get = df.get + return get("person_support_channel") +""", + ), + ( + "dynamic-getattr", + """ +def f(df, attribute): + return getattr(df, attribute) +""", + ), + ( + "late-bound-closure", + """ +def outer(df): + expr = "age >= 18" + def inner(): + return df.query(expr) + expr = "person_support_channel == 1" + return inner() +""", + ), + ), +) +def test_every_review_evasion_is_caught(evasion: str, source: str) -> None: + """Rounds 2-4 are one permanent resolve-or-fail-closed invariant.""" + + assert _source_spine_accesses(source), evasion + + +@pytest.mark.parametrize( + ("control", "source"), + ( + ( + "literal-query", + """ +def f(df): + return df.query("age >= 18") +""", + ), + ( + "bound-query", + """ +def f(df): + expr = "age >= 18" + return df.query(expr) +""", + ), + ( + "resolved-fstring", + """ +def f(df): + column = "age" + return df.query(f"{column} >= 18") +""", + ), + ( + "bare-format", + """ +def f(df): + column = "age" + return df.query("{} >= 18".format(column)) +""", + ), + ( + "named-format", + """ +def f(df): + return df.query("{column} >= 18".format(column="age")) +""", + ), + ( + "bound-filter-list", + """ +def f(df): + columns = ["age", "income"] + return df.filter(items=columns) +""", + ), + ( + "literal-subscript", + """ +def f(df): + return df["age"] +""", + ), + ( + "bound-subscript", + """ +def f(df): + column = "age" + return df[column] +""", + ), + ( + "static-loop", + """ +def f(df): + for column in ("age", "income"): + df[column] +""", + ), + ( + "static-comprehension", + """ +def f(df): + return [df[column] for column in ("age", "income")] +""", + ), + ( + "aliased-query", + """ +def f(df): + query = df.query + return query("age >= 18") +""", + ), + ( + "aliased-eval", + """ +def f(df): + evaluate = df.eval + return evaluate("age + 1") +""", + ), + ( + "aliased-filter", + """ +def f(df): + select = df.filter + return select(items=["age", "income"]) +""", + ), + ( + "aliased-get", + """ +def f(df): + get = df.get + return get("age") +""", + ), + ( + "static-getattr", + """ +def f(df): + return getattr(df, "age") +""", + ), + ( + "single-assignment-closure", + """ +def outer(df): + expr = "age >= 18" + def inner(): + return df.query(expr) + return inner() +""", + ), + ( + "literal-expression-multiplication", + """ +def f(df): + return df.query("age * 2 >= 18") +""", + ), + ( + "typed-row-mask", + """ +def f(values: list[float], mask: list[bool]): + return values[mask] +""", + ), + ( + "typed-mapping-get", + """ +def f(values: dict[str, float], key: str): + return values.get(key) +""", + ), + ), +) +def test_benign_column_access_battery_is_clean(control: str, source: str) -> None: + """Static benign columns and proven non-column access produce no finding.""" + + assert _source_spine_accesses(source) == (), control + + def test_source_spine_ast_guard_covers_every_entity_grain() -> None: """Every US and benefit-unit grain's source identity is prohibited.""" From a0421e9a3e975fa8308cb0fab2f74ecd49967ce6 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 00:33:16 -0400 Subject: [PATCH 43/94] Preserve strictness through composed method aliases --- PROGRESS.md | 16 +- .../tests/test_us_spine_blindness.py | 158 +++++++++++++++--- 2 files changed, 143 insertions(+), 31 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index e6d65412..d063dcff 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -12,7 +12,7 @@ DataFrame `getattr` is fail-closed. Closure free variables now obey lexical late-binding assignment counts. The five requested subsystems, consolidated rounds 2-4 completeness invariant, benign battery, exact graph cardinality, and completed-contract docstrings are implemented. Adversarial audit and full -validation remain. +validation remain. Adversarial alias composition probes are now closed. No push or external mutation is authorized. @@ -78,12 +78,20 @@ No push or external mutation is authorized. - Pinned the multispine tool graph to exactly 54 runtime modules. - Replayed the expanded guard file (70 passed), focused Ruff, and `git diff --check`; the exact 54-module graph remains clean. +- Preserved strict method identity through walrus expressions and structural + tuple/list bindings; a previously strict alias rebound to an unresolved + callable now becomes an explicit opaque alias rather than disappearing. +- Tracked aliases of builtin `getattr` and expanded literal starred arguments; + unresolved starred calls fail closed. +- Added guarded-name, benign, rebound, unpacked, aliased-`getattr`, and starred + `getattr` binding tests. The guard file now passes 71 tests with the graph + clean. ## Next -- Commit the consolidated invariant, benign battery, graph pin, and docstrings. -- Resolve any acceptance-relevant findings from the independent adversarial - audit and replay the guard. +- Commit the alias-composition hardening. +- Resolve starred/nested format fields and closure writes found by the + independent adversarial audit, then replay the guard. - Run the guard file, the full `populace-build` suite, Ruff, and the 54-module graph scan. - Write `/private/tmp/583_fix2_handoff.md` with per-subsystem rules, binding diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index ff8632ea..053a1546 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -780,10 +780,14 @@ def _method_alias( self, node: ast.AST, ) -> tuple[str, bool] | None: + if isinstance(node, ast.NamedExpr): + return self._method_alias(node.value) if isinstance(node, ast.Name): for scope in reversed(self.method_aliases): if node.id in scope: return scope[node.id] + if node.id == "getattr": + return "getattr", True return None if isinstance(node, ast.Attribute) and node.attr in _STRICT_COLUMN_METHODS: strict_opacity = node.attr != "get" or self._attribute_container(node.value) @@ -804,28 +808,53 @@ def _method_alias( return attribute, strict_opacity return None - def _bind(self, targets: list[ast.AST], value: ast.AST) -> None: + def _bind_name(self, name: str, value: ast.AST) -> None: description = self._expression(value) constant: object = _static_string_shape(value, self.constants) if constant is None: constant = _static_string_list(value, self.constants) if constant is None: constant = _static_integer(value, self.constants) - container = self._column_container(value) - attribute_container = self._attribute_container(value) method_alias = self._method_alias(value) - for target in targets: + if method_alias is None and name in self.method_alias_history[-1]: + method_alias = _OPAQUE_METHOD_ALIAS + + self.bindings[-1][name] = description + # None is an explicit opaque shadow: lookups must stop here, + # never fall through to a stale outer binding (shadowed + # parameters and conditional reassignments — sol round 3). + self.constants[-1][name] = constant + self.column_containers[-1][name] = self._column_container(value) + self.attribute_containers[-1][name] = self._attribute_container(value) + self.method_aliases[-1][name] = method_alias + if method_alias is not None: + self.method_alias_history[-1].add(name) + + def _bind_target(self, target: ast.AST, value: ast.AST) -> None: + if isinstance(target, ast.Name): + self._bind_name(target.id, value) + return + if isinstance(target, ast.Starred): + for name in _assigned_names(target.value): + self._bind_name(name, value) + return + if isinstance(target, (ast.List, ast.Tuple)): + if isinstance(value, (ast.List, ast.Tuple)) and len(target.elts) == len( + value.elts + ): + for child_target, child_value in zip( + target.elts, + value.elts, + strict=True, + ): + self._bind_target(child_target, child_value) + return for name in _assigned_names(target): - self.bindings[-1][name] = description - # None is an explicit opaque shadow: lookups must stop here, - # never fall through to a stale outer binding (shadowed - # parameters and conditional reassignments — sol round 3). - self.constants[-1][name] = constant - self.column_containers[-1][name] = container - self.attribute_containers[-1][name] = attribute_container - self.method_aliases[-1][name] = method_alias - if method_alias is not None: - self.method_alias_history[-1].add(name) + self._bind_name(name, value) + + def _bind(self, targets: list[ast.AST], value: ast.AST) -> None: + for target in targets: + self._bind_target(target, value) def visit_Assign(self, node: ast.Assign) -> None: self.visit(node.value) @@ -1137,11 +1166,28 @@ def _poison(self, name: str) -> None: self.constants[-1][name] = None def _visit_getattr(self, node: ast.Call) -> None: - if len(node.args) < 2: + arguments: list[ast.AST] = [] + for argument in node.args: + if isinstance(argument, ast.Starred): + if not isinstance(argument.value, (ast.List, ast.Tuple)): + self._record( + node, + "getattr() with hidden or expanded arguments (fail-closed)", + ) + return + arguments.extend(argument.value.elts) + else: + arguments.append(argument) + if node.keywords or len(arguments) < 2: + if node.args or node.keywords: + self._record( + node, + "getattr() with hidden or expanded arguments (fail-closed)", + ) return - attribute = _static_string_shape(node.args[1], self.constants) + attribute = _static_string_shape(arguments[1], self.constants) if attribute is None or _OPAQUE_STRING_PART in attribute: - if self._attribute_container(node.args[0]): + if self._attribute_container(arguments[0]): self._record( node, "getattr with an unresolvable dynamic attribute (fail-closed)", @@ -1271,7 +1317,9 @@ def _visit_strict_method_call( method: str, strict_opacity: bool, ) -> None: - if method == "get": + if method == "getattr": + self._visit_getattr(node) + elif method == "get": self._visit_get_call(node, strict_opacity=strict_opacity) elif method in {"query", "eval"}: self._visit_query_or_eval_call(node, method=method) @@ -2149,7 +2197,7 @@ def f(df): def test_strict_method_aliases_fail_closed_and_shadow_precisely() -> None: - """Opaque alias arguments fail; rebinding and parameters stop stale aliases.""" + """Opaque alias arguments/rebindings fail; parameters stop stale aliases.""" opaque_sources = ( """ @@ -2178,25 +2226,59 @@ def f(df, kwargs): return query(**kwargs) """, ) - shadowed_sources = ( - """ + rebound_source = """ def f(df): query = df.query query = print return query("age >= 18") -""", - """ +""" + parameter_shadow = """ query = df.query def f(query): return query("age >= 18") -""", - ) +""" for source in opaque_sources: accesses = _source_spine_accesses(source) assert accesses, source assert any("fail-closed" in item for item in accesses) - for source in shadowed_sources: + rebound_accesses = _source_spine_accesses(rebound_source) + assert rebound_accesses + assert any("fail-closed" in item for item in rebound_accesses) + assert _source_spine_accesses(parameter_shadow) == () + + +def test_strict_method_aliases_cover_expression_and_structural_bindings() -> None: + """Walrus and unpacked aliases retain strict method identity.""" + + guarded_sources = ( + """ +def f(df): + return (query := df.query)("person_support_channel == 1") +""", + """ +def f(df): + query, printer = (df.query, print) + return query("person_support_channel == 1") +""", + ) + benign_sources = ( + """ +def f(df): + return (query := df.query)("age >= 18") +""", + """ +def f(df): + query, printer = (df.query, print) + return query("age >= 18") +""", + ) + + for source in guarded_sources: + accesses = _source_spine_accesses(source) + assert accesses, source + assert any("person_support_channel" in item for item in accesses) + for source in benign_sources: assert _source_spine_accesses(source) == (), source @@ -2219,6 +2301,19 @@ def f(df): dynamic_attribute = """ def f(df, attribute): return getattr(df, attribute) +""" + aliased_getattr = """ +def f(df): + lookup = getattr + return lookup(df, "person_support_channel") +""" + starred_getattr = """ +def f(df): + return getattr(*(df, "person_support_channel")) +""" + opaque_starred_getattr = """ +def f(arguments): + return getattr(*arguments) """ benign_attribute = """ def f(df): @@ -2230,13 +2325,22 @@ def f(obj: object, attribute: str): return getattr(obj, attribute) """ - for source in (static_alias, immediate_alias, guarded_attribute): + for source in ( + static_alias, + immediate_alias, + guarded_attribute, + aliased_getattr, + starred_getattr, + ): accesses = _source_spine_accesses(source) assert accesses, source assert any("person_support_channel" in item for item in accesses) dynamic_accesses = _source_spine_accesses(dynamic_attribute) assert dynamic_accesses assert any("fail-closed" in item for item in dynamic_accesses) + opaque_starred_accesses = _source_spine_accesses(opaque_starred_getattr) + assert opaque_starred_accesses + assert any("fail-closed" in item for item in opaque_starred_accesses) assert _source_spine_accesses(benign_attribute) == () assert _source_spine_accesses(generic_object) == () From 7da96e68ded43474ee1a08c9df4df280493f5d19 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 00:35:06 -0400 Subject: [PATCH 44/94] Resolve composed static format fields --- PROGRESS.md | 16 +- .../tests/test_us_spine_blindness.py | 232 ++++++++++++++---- 2 files changed, 196 insertions(+), 52 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index d063dcff..9c312ab1 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -13,6 +13,7 @@ late-binding assignment counts. The five requested subsystems, consolidated rounds 2-4 completeness invariant, benign battery, exact graph cardinality, and completed-contract docstrings are implemented. Adversarial audit and full validation remain. Adversarial alias composition probes are now closed. +Starred and nested format-field composition is now closed too. No push or external mutation is authorized. @@ -86,12 +87,21 @@ No push or external mutation is authorized. - Added guarded-name, benign, rebound, unpacked, aliased-`getattr`, and starred `getattr` binding tests. The guard file now passes 71 tests with the graph clean. +- Expanded statically known `*args`/`**kwargs` before assigning format field + positions, while any unresolved expansion that can supply a referenced field + becomes opaque. +- Recursively resolved nested format specifications and static collection + indexing, and resolved static mapping operands for percent formatting. +- Added exact-name and benign controls for nested specs, indexed mappings, + static positional/keyword expansion, and percent mappings, plus the + adversarial star-index-shift failure. The guard remains 71 tests green with + the graph clean. ## Next -- Commit the alias-composition hardening. -- Resolve starred/nested format fields and closure writes found by the - independent adversarial audit, then replay the guard. +- Commit complete composed format resolution. +- Resolve closure writes found by the independent adversarial audit, then + replay the guard. - Run the guard file, the full `populace-build` suite, Ruff, and the 54-module graph scan. - Write `/private/tmp/583_fix2_handoff.md` with per-subsystem rules, binding diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 053a1546..c831ccf2 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -452,11 +452,50 @@ def _static_format_value( integer = _static_integer(node, constants) if integer is not None: return integer + return _static_literal_value(node, constants) + + +def _static_literal_value( + node: ast.AST, + constants: list[dict[str, object]], +) -> object: + if isinstance(node, ast.Name): + for scope in reversed(constants): + if node.id in scope: + value = scope[node.id] + return _OPAQUE_STATIC_VALUE if value is None else value + return _OPAQUE_STATIC_VALUE if isinstance(node, ast.Constant) and isinstance( node.value, - (float, bool, bytes, type(None)), + (str, int, float, bool, bytes, type(None)), ): return node.value + if isinstance(node, (ast.List, ast.Tuple)): + values = tuple( + _static_literal_value(element, constants) for element in node.elts + ) + if any(value is _OPAQUE_STATIC_VALUE for value in values): + return _OPAQUE_STATIC_VALUE + return list(values) if isinstance(node, ast.List) else values + if isinstance(node, ast.Dict): + if any(key is None for key in node.keys): + return _OPAQUE_STATIC_VALUE + keys = tuple( + _static_literal_value(key, constants) + for key in node.keys + if key is not None + ) + values = tuple( + _static_literal_value(value, constants) for value in node.values + ) + if any( + value is _OPAQUE_STATIC_VALUE for value in (*keys, *values) + ): + return _OPAQUE_STATIC_VALUE + try: + return dict(zip(keys, values, strict=True)) + except TypeError: + return _OPAQUE_STATIC_VALUE return _OPAQUE_STATIC_VALUE @@ -468,54 +507,84 @@ def _resolve_static_format( """Substitute every statically known ``str.format`` field.""" formatter = Formatter() - positional = tuple( - _OPAQUE_STATIC_VALUE - if isinstance(argument, ast.Starred) - else _static_format_value(argument, constants) - for argument in node.args - ) - keywords = { - keyword.arg: _static_format_value(keyword.value, constants) - for keyword in node.keywords - if keyword.arg is not None - } - expanded_keywords = any(keyword.arg is None for keyword in node.keywords) - auto_index = 0 - pieces: list[str] = [] - try: - parsed = tuple(formatter.parse(template)) - except ValueError: - return _OPAQUE_STRING_PART - for literal, field_name, format_spec, conversion in parsed: - pieces.append(literal) - if field_name is None: - continue - lookup_name = field_name - if field_name == "": - lookup_name = str(auto_index) - auto_index += 1 - try: - value, _ = formatter.get_field( - lookup_name, - positional, - keywords, - ) - except (AttributeError, IndexError, KeyError, TypeError, ValueError): - value = _OPAQUE_STATIC_VALUE - if value is _OPAQUE_STATIC_VALUE or ( - expanded_keywords and not lookup_name.isdecimal() - ): - pieces.append(_OPAQUE_STRING_PART) + positional: list[object] = [] + expanded_positional = False + for argument in node.args: + if not isinstance(argument, ast.Starred): + positional.append(_static_format_value(argument, constants)) continue - if "{" in format_spec or "}" in format_spec: - pieces.append(_OPAQUE_STRING_PART) + expanded = _static_literal_value(argument.value, constants) + if isinstance(expanded, (list, tuple)): + positional.extend(expanded) + else: + expanded_positional = True + + keywords: dict[str, object] = {} + expanded_keywords = False + for keyword in node.keywords: + if keyword.arg is not None: + keywords[keyword.arg] = _static_format_value(keyword.value, constants) continue + expanded = _static_literal_value(keyword.value, constants) + if isinstance(expanded, dict) and all( + isinstance(key, str) for key in expanded + ): + keywords.update(expanded) + else: + expanded_keywords = True + + auto_index = [0] + + def resolve_fields(value: str) -> str: + pieces: list[str] = [] try: - converted = formatter.convert_field(value, conversion) - pieces.append(formatter.format_field(converted, format_spec)) - except (TypeError, ValueError): - pieces.append(_OPAQUE_STRING_PART) - return "".join(pieces) + parsed = tuple(formatter.parse(value)) + except ValueError: + return _OPAQUE_STRING_PART + for literal, field_name, format_spec, conversion in parsed: + pieces.append(literal) + if field_name is None: + continue + lookup_name = field_name + if field_name == "": + lookup_name = str(auto_index[0]) + auto_index[0] += 1 + root_name = re.split(r"[.[]", lookup_name, maxsplit=1)[0] + if ( + expanded_positional + and root_name.isdecimal() + or expanded_keywords + and not root_name.isdecimal() + and root_name not in keywords + ): + pieces.append(_OPAQUE_STRING_PART) + continue + try: + field_value, _ = formatter.get_field( + lookup_name, + tuple(positional), + keywords, + ) + except (AttributeError, IndexError, KeyError, TypeError, ValueError): + field_value = _OPAQUE_STATIC_VALUE + if field_value is _OPAQUE_STATIC_VALUE: + pieces.append(_OPAQUE_STRING_PART) + continue + resolved_spec = resolve_fields(format_spec) if format_spec else "" + if _OPAQUE_STRING_PART in resolved_spec: + pieces.append(_OPAQUE_STRING_PART) + continue + try: + converted = formatter.convert_field(field_value, conversion) + pieces.append(formatter.format_field(converted, resolved_spec)) + except (TypeError, ValueError): + pieces.append(_OPAQUE_STRING_PART) + return "".join(pieces) + + try: + return resolve_fields(template) + except (IndexError, KeyError, ValueError): + return _OPAQUE_STRING_PART def _static_integer(node: ast.AST, constants: list[dict[str, object]]) -> int | None: @@ -543,7 +612,7 @@ def _static_percent_operand( for scope in reversed(constants): if node.id in scope: value = scope[node.id] - if isinstance(value, (str, int, float, bytes, tuple)): + if isinstance(value, (str, int, float, bytes, tuple, dict)): return value return _OPAQUE_STATIC_VALUE return _OPAQUE_STATIC_VALUE @@ -556,6 +625,9 @@ def _static_percent_operand( if any(value is _OPAQUE_STATIC_VALUE for value in values): return _OPAQUE_STATIC_VALUE return values + if isinstance(node, ast.Dict): + value = _static_literal_value(node, constants) + return value if isinstance(value, dict) else _OPAQUE_STATIC_VALUE return _OPAQUE_STATIC_VALUE @@ -568,10 +640,10 @@ def _static_string_list( for scope in reversed(constants): if node.id in scope: value = scope[node.id] - if isinstance(value, tuple) and all( + if isinstance(value, (list, tuple)) and all( isinstance(item, str) for item in value ): - return value + return tuple(value) return None return None if isinstance(node, (ast.List, ast.Tuple)): @@ -815,6 +887,10 @@ def _bind_name(self, name: str, value: ast.AST) -> None: constant = _static_string_list(value, self.constants) if constant is None: constant = _static_integer(value, self.constants) + if constant is None: + literal = _static_literal_value(value, self.constants) + if literal is not _OPAQUE_STATIC_VALUE: + constant = literal method_alias = self._method_alias(value) if method_alias is None and name in self.method_alias_history[-1]: method_alias = _OPAQUE_METHOD_ALIAS @@ -2096,6 +2172,40 @@ def f(df): return df[ "{entity}_support_channel".format(entity="person") ] +""", + """ +def f(df): + return df.query( + "{0:{1}} == 1".format("person_support_channel", "s") + ) +""", + """ +def f(df): + return df.query( + "{0[column]} == 1".format( + {"column": "person_support_channel"} + ) + ) +""", + """ +def f(df): + return df.query( + "{} == 1".format(*("person_support_channel",)) + ) +""", + """ +def f(df): + return df.query( + "{column} == 1".format( + **{"column": "person_support_channel"} + ) + ) +""", + """ +def f(df): + return df.query( + "%(entity)s_support_channel == 1" % {"entity": "person"} + ) """, ) @@ -2125,6 +2235,26 @@ def f(df): """ def f(df): return df.query("{{age}} == {{age}}".format()) +""", + """ +def f(df): + return df.query("{0:{1}} >= 18".format("age", "s")) +""", + """ +def f(df): + return df.query("{0[column]} >= 18".format({"column": "age"})) +""", + """ +def f(df): + return df.query("{} >= 18".format(*("age",))) +""", + """ +def f(df): + return df.query("{column} >= 18".format(**{"column": "age"})) +""", + """ +def f(df): + return df.query("%(column)s >= 18" % {"column": "age"}) """, ) opaque_sources = ( @@ -2143,6 +2273,10 @@ def f(df, values): """ def f(df, values): return df.query("{col} == 1".format(**values)) +""", + """ +def f(df, values): + return df.query("{1} == 1".format(*values, "age")) """, ) From 4df316a225b84df5e6ef27c9ed455d77ea136007 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 00:37:38 -0400 Subject: [PATCH 45/94] Count late writes across closure boundaries --- PROGRESS.md | 18 +- .../tests/test_us_spine_blindness.py | 192 ++++++++++++++++-- 2 files changed, 184 insertions(+), 26 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 9c312ab1..7fd90f8d 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -13,7 +13,8 @@ late-binding assignment counts. The five requested subsystems, consolidated rounds 2-4 completeness invariant, benign battery, exact graph cardinality, and completed-contract docstrings are implemented. Adversarial audit and full validation remain. Adversarial alias composition probes are now closed. -Starred and nested format-field composition is now closed too. +Starred and nested format-field composition is now closed too, as are +late-bound writes through comprehensions, `nonlocal`, and `global`. No push or external mutation is authorized. @@ -96,12 +97,21 @@ No push or external mutation is authorized. static positional/keyword expansion, and percent mappings, plus the adversarial star-index-shift failure. The guard remains 71 tests green with the graph clean. +- Counted assignment-expression targets inside comprehensions in their actual + containing Python scope and routed their runtime bindings past the synthetic + comprehension scope. +- Counted nested `nonlocal`/`global` stores against the lexical scope they can + rebind, while excluding those declarations from the nested function's local + binding count. +- Added direct subscript and closure tests for comprehension walruses plus + sibling `nonlocal` and module `global` writers. The guard remains 71 tests + green with the graph clean. ## Next -- Commit complete composed format resolution. -- Resolve closure writes found by the independent adversarial audit, then - replay the guard. +- Commit closure write composition. +- Reconcile the universal column-container contract with the typed runtime + graph findings from the independent audit. - Run the guard file, the full `populace-build` suite, Ruff, and the 54-module graph scan. - Write `/private/tmp/583_fix2_handoff.md` with per-subsystem rules, binding diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index c831ccf2..58b4ec93 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -720,6 +720,7 @@ class _ScopeAssignmentCounter(ast.NodeVisitor): def __init__(self) -> None: self.counts: dict[str, int] = {} + self.external_names: set[str] = set() def _count(self, name: str) -> None: self.counts[name] = self.counts.get(name, 0) + 1 @@ -773,27 +774,89 @@ def visit_ExceptHandler(self, node: ast.ExceptHandler) -> None: for statement in node.body: self.visit(statement) + def visit_Global(self, node: ast.Global) -> None: + self.external_names.update(node.names) + + def visit_Nonlocal(self, node: ast.Nonlocal) -> None: + self.external_names.update(node.names) + + def _count_comprehension_named_expressions( + self, + node: ast.ListComp | ast.SetComp | ast.DictComp | ast.GeneratorExp, + ) -> None: + for descendant in ast.walk(node): + if isinstance(descendant, ast.NamedExpr): + for name in _assigned_names(descendant.target): + self._count(name) + def visit_ListComp(self, node: ast.ListComp) -> None: - self.visit(node.generators[0].iter) + self._count_comprehension_named_expressions(node) def visit_SetComp(self, node: ast.SetComp) -> None: - self.visit(node.generators[0].iter) + self._count_comprehension_named_expressions(node) def visit_DictComp(self, node: ast.DictComp) -> None: - self.visit(node.generators[0].iter) + self._count_comprehension_named_expressions(node) def visit_GeneratorExp(self, node: ast.GeneratorExp) -> None: - self.visit(node.generators[0].iter) + self._count_comprehension_named_expressions(node) + + +def _nested_external_writes( + body: list[ast.stmt], + declaration: type[ast.Global] | type[ast.Nonlocal], +) -> set[str]: + class NestedWriteCollector(ast.NodeVisitor): + def __init__(self) -> None: + self.depth = 0 + self.declared: set[str] = set() + self.stored: set[str] = set() + + def _visit_function( + self, + node: ast.FunctionDef | ast.AsyncFunctionDef, + ) -> None: + self.depth += 1 + self.generic_visit(node) + self.depth -= 1 + + def visit_FunctionDef(self, node: ast.FunctionDef) -> None: + self._visit_function(node) + + def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None: + self._visit_function(node) + + def visit_Global(self, node: ast.Global) -> None: + if self.depth and declaration is ast.Global: + self.declared.update(node.names) + + def visit_Nonlocal(self, node: ast.Nonlocal) -> None: + if self.depth and declaration is ast.Nonlocal: + self.declared.update(node.names) + + def visit_Name(self, node: ast.Name) -> None: + if self.depth and isinstance(node.ctx, ast.Store): + self.stored.add(node.id) + + collector = NestedWriteCollector() + for statement in body: + collector.visit(statement) + return collector.declared & collector.stored def _scope_assignment_counts( body: list[ast.stmt], *, parameters: tuple[str, ...] = (), + nested_declaration: type[ast.Global] | type[ast.Nonlocal] = ast.Nonlocal, ) -> dict[str, int]: counter = _ScopeAssignmentCounter() for statement in body: counter.visit(statement) + for name in counter.external_names: + counter.counts.pop(name, None) + for name in _nested_external_writes(body, nested_declaration): + counter._count(name) for name in parameters: counter._count(name) return counter.counts @@ -809,6 +872,7 @@ def __init__(self, factory_aliases: set[str]) -> None: self.method_aliases: list[dict[str, tuple[str, bool] | None]] = [{}] self.method_alias_history: list[set[str]] = [set()] self.assignment_counts: list[dict[str, int]] = [{}] + self.scope_kinds = ["module"] self.accesses: set[tuple[int, int, str]] = set() def _expression(self, node: ast.AST) -> str | None: @@ -880,7 +944,13 @@ def _method_alias( return attribute, strict_opacity return None - def _bind_name(self, name: str, value: ast.AST) -> None: + def _bind_name( + self, + name: str, + value: ast.AST, + *, + scope_index: int = -1, + ) -> None: description = self._expression(value) constant: object = _static_string_shape(value, self.constants) if constant is None: @@ -892,27 +962,33 @@ def _bind_name(self, name: str, value: ast.AST) -> None: if literal is not _OPAQUE_STATIC_VALUE: constant = literal method_alias = self._method_alias(value) - if method_alias is None and name in self.method_alias_history[-1]: + if method_alias is None and name in self.method_alias_history[scope_index]: method_alias = _OPAQUE_METHOD_ALIAS - self.bindings[-1][name] = description + self.bindings[scope_index][name] = description # None is an explicit opaque shadow: lookups must stop here, # never fall through to a stale outer binding (shadowed # parameters and conditional reassignments — sol round 3). - self.constants[-1][name] = constant - self.column_containers[-1][name] = self._column_container(value) - self.attribute_containers[-1][name] = self._attribute_container(value) - self.method_aliases[-1][name] = method_alias + self.constants[scope_index][name] = constant + self.column_containers[scope_index][name] = self._column_container(value) + self.attribute_containers[scope_index][name] = self._attribute_container(value) + self.method_aliases[scope_index][name] = method_alias if method_alias is not None: - self.method_alias_history[-1].add(name) + self.method_alias_history[scope_index].add(name) - def _bind_target(self, target: ast.AST, value: ast.AST) -> None: + def _bind_target( + self, + target: ast.AST, + value: ast.AST, + *, + scope_index: int = -1, + ) -> None: if isinstance(target, ast.Name): - self._bind_name(target.id, value) + self._bind_name(target.id, value, scope_index=scope_index) return if isinstance(target, ast.Starred): for name in _assigned_names(target.value): - self._bind_name(name, value) + self._bind_name(name, value, scope_index=scope_index) return if isinstance(target, (ast.List, ast.Tuple)): if isinstance(value, (ast.List, ast.Tuple)) and len(target.elts) == len( @@ -923,14 +999,24 @@ def _bind_target(self, target: ast.AST, value: ast.AST) -> None: value.elts, strict=True, ): - self._bind_target(child_target, child_value) + self._bind_target( + child_target, + child_value, + scope_index=scope_index, + ) return for name in _assigned_names(target): - self._bind_name(name, value) + self._bind_name(name, value, scope_index=scope_index) - def _bind(self, targets: list[ast.AST], value: ast.AST) -> None: + def _bind( + self, + targets: list[ast.AST], + value: ast.AST, + *, + scope_index: int = -1, + ) -> None: for target in targets: - self._bind_target(target, value) + self._bind_target(target, value, scope_index=scope_index) def visit_Assign(self, node: ast.Assign) -> None: self.visit(node.value) @@ -963,7 +1049,12 @@ def visit_AnnAssign(self, node: ast.AnnAssign) -> None: def visit_NamedExpr(self, node: ast.NamedExpr) -> None: self.visit(node.value) - self._bind([node.target], node.value) + scope_index = next( + index + for index in range(len(self.scope_kinds) - 1, -1, -1) + if self.scope_kinds[index] != "comprehension" + ) + self._bind([node.target], node.value, scope_index=scope_index) def _bind_iteration_target( self, @@ -976,6 +1067,8 @@ def _bind_iteration_target( self.column_containers[-1][name] = False self.attribute_containers[-1][name] = False self.method_aliases[-1][name] = None + if self.scope_kinds[-1] == "comprehension": + self.assignment_counts[-1][name] = 1 def visit_For(self, node: ast.For) -> None: self.visit(node.iter) @@ -1000,6 +1093,7 @@ def _visit_comprehension( self.method_aliases.append({}) self.method_alias_history.append(set()) self.assignment_counts.append({}) + self.scope_kinds.append("comprehension") for generator in node.generators: self.visit(generator.iter) self._bind_iteration_target( @@ -1014,6 +1108,7 @@ def _visit_comprehension( else: self.visit(node.elt) self.assignment_counts.pop() + self.scope_kinds.pop() self.method_alias_history.pop() self.method_aliases.pop() self.attribute_containers.pop() @@ -1047,7 +1142,10 @@ def _visit_scope_statements(self, body: list[ast.stmt]) -> None: self.visit(function) def visit_Module(self, node: ast.Module) -> None: - counts = _scope_assignment_counts(node.body) + counts = _scope_assignment_counts( + node.body, + nested_declaration=ast.Global, + ) self.assignment_counts[0] = counts for name in counts: self.bindings[0].setdefault(name, None) @@ -1145,7 +1243,9 @@ def _visit_function( self.method_aliases.append(method_aliases) self.method_alias_history.append(alias_history) self.assignment_counts.append(counts) + self.scope_kinds.append("function") self._visit_scope_statements(node.body) + self.scope_kinds.pop() self.assignment_counts.pop() self.method_alias_history.pop() self.method_aliases.pop() @@ -1214,7 +1314,9 @@ def visit_Lambda(self, node: ast.Lambda) -> None: self.method_aliases.append(method_aliases) self.method_alias_history.append(alias_history) self.assignment_counts.append(counts) + self.scope_kinds.append("lambda") self.visit(node.body) + self.scope_kinds.pop() self.assignment_counts.pop() self.method_alias_history.pop() self.method_aliases.pop() @@ -1994,10 +2096,21 @@ def f(df): .replace("SOURCE", "support") ] """ + comprehension_walrus = """ +def f(df): + col = "age" + [(col := "person_support_channel") for _ in (0,)] + return df[col] +""" walrus_accesses = _source_spine_accesses(walrus) assert len(walrus_accesses) == 2 assert all("person_support_channel" in access for access in walrus_accesses) + comprehension_accesses = _source_spine_accesses(comprehension_walrus) + assert comprehension_accesses + assert all( + "person_support_channel" in access for access in comprehension_accesses + ) for source in (multiplication, percent_format, replace_chain): accesses = _source_spine_accesses(source) @@ -2525,8 +2638,43 @@ def outer(df): expr = "person_support_channel == 1" return inner() """ + comprehension_rebound = """ +def outer(df): + expr = "age >= 18" + def inner(): + return df.query(expr) + [(expr := "person_support_channel == 1") for _ in (0,)] + return inner() +""" + nonlocal_rebound = """ +def outer(df): + expr = "age >= 18" + def rebind(): + nonlocal expr + expr = "person_support_channel == 1" + def inner(): + return df.query(expr) + rebind() + return inner() +""" + global_rebound = """ +expr = "age >= 18" +def rebind(): + global expr + expr = "person_support_channel == 1" +def f(df): + rebind() + return df.query(expr) +""" - for source in (late_bound, module_late_bound, lambda_late_bound): + for source in ( + late_bound, + module_late_bound, + lambda_late_bound, + comprehension_rebound, + nonlocal_rebound, + global_rebound, + ): accesses = _source_spine_accesses(source) assert accesses, source assert any("fail-closed" in item for item in accesses) From a2e4d87699cfdc1aafc0563992e7bedeb8be14ef Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 00:38:42 -0400 Subject: [PATCH 46/94] Propagate every static string iterable form --- PROGRESS.md | 9 +- .../tests/test_us_spine_blindness.py | 104 ++++++++++++++++-- 2 files changed, 105 insertions(+), 8 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 7fd90f8d..3e39283b 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -15,6 +15,8 @@ and completed-contract docstrings are implemented. Adversarial audit and full validation remain. Adversarial alias composition probes are now closed. Starred and nested format-field composition is now closed too, as are late-bound writes through comprehensions, `nonlocal`, and `global`. +Static iterable propagation now covers all built-in literal forms requested by +the contract. No push or external mutation is authorized. @@ -106,10 +108,15 @@ No push or external mutation is authorized. - Added direct subscript and closure tests for comprehension walruses plus sibling `nonlocal` and module `global` writers. The guard remains 71 tests green with the graph clean. +- Generalized static string iteration from list/tuple syntax to sets, dict + keys, literal strings, and statically concatenated collections while + preserving literal containers for format-field indexing. +- Added benign and guarded loop/comprehension controls for every new iterable + form. The guard file now passes 72 tests with the graph clean. ## Next -- Commit closure write composition. +- Commit complete static-iterable propagation. - Reconcile the universal column-container contract with the typed runtime graph findings from the independent audit. - Run the guard file, the full `populace-build` suite, Ruff, and the 54-module diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 58b4ec93..44386778 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -470,13 +470,20 @@ def _static_literal_value( (str, int, float, bool, bytes, type(None)), ): return node.value - if isinstance(node, (ast.List, ast.Tuple)): + if isinstance(node, (ast.List, ast.Set, ast.Tuple)): values = tuple( _static_literal_value(element, constants) for element in node.elts ) if any(value is _OPAQUE_STATIC_VALUE for value in values): return _OPAQUE_STATIC_VALUE - return list(values) if isinstance(node, ast.List) else values + if isinstance(node, ast.List): + return list(values) + if isinstance(node, ast.Set): + try: + return set(values) + except TypeError: + return _OPAQUE_STATIC_VALUE + return values if isinstance(node, ast.Dict): if any(key is None for key in node.keys): return _OPAQUE_STATIC_VALUE @@ -634,19 +641,28 @@ def _static_percent_operand( def _static_string_list( node: ast.AST, constants: list[dict[str, object]] ) -> tuple[str, ...] | None: - """Resolve a static list/tuple of strings, through one Name binding.""" + """Resolve any statically enumerable string iterable.""" if isinstance(node, ast.Name): for scope in reversed(constants): if node.id in scope: value = scope[node.id] - if isinstance(value, (list, tuple)) and all( + if isinstance(value, dict) and all( isinstance(item, str) for item in value ): return tuple(value) + if isinstance(value, (list, set, tuple)) and all( + isinstance(item, str) for item in value + ): + return tuple(value) + if isinstance(value, str): + return tuple(value) return None return None - if isinstance(node, (ast.List, ast.Tuple)): + literal = _literal_string(node) + if literal is not None: + return tuple(literal) + if isinstance(node, (ast.List, ast.Set, ast.Tuple)): items: list[str] = [] for element in node.elts: values = _static_string_values(element, constants) @@ -654,6 +670,22 @@ def _static_string_list( return None items.extend(values) return tuple(items) + if isinstance(node, ast.Dict): + items: list[str] = [] + for key in node.keys: + if key is None: + return None + values = _static_string_values(key, constants) + if values is None: + return None + items.extend(values) + return tuple(items) + if isinstance(node, ast.BinOp) and isinstance(node.op, ast.Add): + left = _static_string_list(node.left, constants) + right = _static_string_list(node.right, constants) + if left is not None and right is not None: + return (*left, *right) + return None if isinstance(node, (ast.ListComp, ast.SetComp, ast.GeneratorExp)): local: dict[str, object] = {} nested_constants = [*constants, local] @@ -953,14 +985,14 @@ def _bind_name( ) -> None: description = self._expression(value) constant: object = _static_string_shape(value, self.constants) - if constant is None: - constant = _static_string_list(value, self.constants) if constant is None: constant = _static_integer(value, self.constants) if constant is None: literal = _static_literal_value(value, self.constants) if literal is not _OPAQUE_STATIC_VALUE: constant = literal + if constant is None: + constant = _static_string_list(value, self.constants) method_alias = self._method_alias(value) if method_alias is None and name in self.method_alias_history[scope_index]: method_alias = _OPAQUE_METHOD_ALIAS @@ -2190,6 +2222,64 @@ def f(df, columns): assert all("fail-closed" in item for item in dynamic_accesses) +def test_loop_targets_accept_every_static_string_iterable_form() -> None: + """Sets, dict keys, strings, and concatenation bind exact choices.""" + + benign_sources = ( + """ +def f(df): + for col in {"age", "income"}: + df[col] +""", + """ +def f(df): + for col in {"age": 1, "income": 2}: + df[col] +""", + """ +def f(df): + for col in "ab": + df[col] +""", + """ +def f(df): + for col in ("age",) + ("income",): + df[col] +""", + """ +def f(df): + return [df[col] for col in {"age", "income"}] +""", + ) + guarded_sources = ( + """ +def f(df): + for col in {"age", "person_support_channel"}: + df[col] +""", + """ +def f(df): + for col in {"person_support_channel": 1, "age": 2}: + df[col] +""", + """ +def f(df): + return [ + df[col] + for col in ("age",) + ("person_support_channel",) + ] +""", + ) + + for source in benign_sources: + assert _source_spine_accesses(source) == (), source + for source in guarded_sources: + accesses = _source_spine_accesses(source) + assert accesses, source + assert any("person_support_channel" in item for item in accesses) + assert all("fail-closed" not in item for item in accesses) + + def test_comprehension_targets_propagate_static_and_opaque_choices() -> None: """All comprehension forms bind targets before visiting their bodies.""" From 1f7405e2eb870c2a359e8f0e77d25a0e0aa46399 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 00:40:06 -0400 Subject: [PATCH 47/94] Join conditional selector states conservatively --- PROGRESS.md | 11 +- .../tests/test_us_spine_blindness.py | 225 +++++++++++++++++- 2 files changed, 232 insertions(+), 4 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 3e39283b..4926f824 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -17,6 +17,8 @@ Starred and nested format-field composition is now closed too, as are late-bound writes through comprehensions, `nonlocal`, and `global`. Static iterable propagation now covers all built-in literal forms requested by the contract. +Conditional statements and optional loop execution now merge bindings +conservatively instead of retaining the last syntactically visited value. No push or external mutation is authorized. @@ -113,10 +115,17 @@ No push or external mutation is authorized. preserving literal containers for format-field indexing. - Added benign and guarded loop/comprehension controls for every new iterable form. The guard file now passes 72 tests with the graph clean. +- Added branch-state joins for `if` statements/expressions and zero-or-more + iteration joins for dynamic loops; divergent constants become opaque, + possible DataFrame provenance is retained, and conditional method aliases + cannot disappear. +- Preserved exact pre-loop bindings across provably empty static iterables. +- Added guarded, benign, conditional-alias, optional-loop, and empty-loop + binding tests. The guard file now passes 73 tests with the graph clean. ## Next -- Commit complete static-iterable propagation. +- Commit flow-sensitive branch and loop joins. - Reconcile the universal column-container contract with the typed runtime graph findings from the independent audit. - Run the guard file, the full `populace-build` suite, Ruff, and the 54-module diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 44386778..dea6c671 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -1098,17 +1098,177 @@ def _bind_iteration_target( self.constants[-1][name] = values self.column_containers[-1][name] = False self.attribute_containers[-1][name] = False - self.method_aliases[-1][name] = None + self.method_aliases[-1][name] = ( + _OPAQUE_METHOD_ALIAS + if name in self.method_alias_history[-1] + else None + ) if self.scope_kinds[-1] == "comprehension": self.assignment_counts[-1][name] = 1 + def _flow_state( + self, + ) -> tuple[ + dict[str, str | None], + dict[str, object], + dict[str, bool], + dict[str, bool], + dict[str, tuple[str, bool] | None], + set[str], + ]: + return ( + self.bindings[-1].copy(), + self.constants[-1].copy(), + self.column_containers[-1].copy(), + self.attribute_containers[-1].copy(), + self.method_aliases[-1].copy(), + self.method_alias_history[-1].copy(), + ) + + def _restore_flow_state( + self, + state: tuple[ + dict[str, str | None], + dict[str, object], + dict[str, bool], + dict[str, bool], + dict[str, tuple[str, bool] | None], + set[str], + ], + ) -> None: + ( + self.bindings[-1], + self.constants[-1], + self.column_containers[-1], + self.attribute_containers[-1], + self.method_aliases[-1], + self.method_alias_history[-1], + ) = tuple(value.copy() for value in state) + + @staticmethod + def _same_flow_value(left: object, right: object) -> bool: + return type(left) is type(right) and left == right + + def _merge_flow_states( + self, + left: tuple[ + dict[str, str | None], + dict[str, object], + dict[str, bool], + dict[str, bool], + dict[str, tuple[str, bool] | None], + set[str], + ], + right: tuple[ + dict[str, str | None], + dict[str, object], + dict[str, bool], + dict[str, bool], + dict[str, tuple[str, bool] | None], + set[str], + ], + ) -> None: + left_bindings, left_constants, left_columns, left_attributes, left_aliases, left_history = left + ( + right_bindings, + right_constants, + right_columns, + right_attributes, + right_aliases, + right_history, + ) = right + names = ( + set(left_bindings) + | set(right_bindings) + | set(left_constants) + | set(right_constants) + | set(left_aliases) + | set(right_aliases) + ) + history = left_history | right_history + self.bindings[-1] = { + name: ( + left_bindings.get(name) + if self._same_flow_value( + left_bindings.get(name), + right_bindings.get(name), + ) + else None + ) + for name in names + } + self.constants[-1] = { + name: ( + left_constants.get(name) + if self._same_flow_value( + left_constants.get(name), + right_constants.get(name), + ) + else None + ) + for name in names + } + self.column_containers[-1] = { + name: left_columns.get(name, False) or right_columns.get(name, False) + for name in names + } + self.attribute_containers[-1] = { + name: left_attributes.get(name, False) + or right_attributes.get(name, False) + for name in names + } + self.method_aliases[-1] = {} + for name in names: + left_alias = left_aliases.get(name) + right_alias = right_aliases.get(name) + if self._same_flow_value(left_alias, right_alias): + merged_alias = left_alias + elif name in history or left_alias is not None or right_alias is not None: + merged_alias = _OPAQUE_METHOD_ALIAS + history.add(name) + else: + merged_alias = None + self.method_aliases[-1][name] = merged_alias + self.method_alias_history[-1] = history + + def visit_If(self, node: ast.If) -> None: + self.visit(node.test) + before = self._flow_state() + for statement in node.body: + self.visit(statement) + body_state = self._flow_state() + self._restore_flow_state(before) + for statement in node.orelse: + self.visit(statement) + else_state = self._flow_state() + self._merge_flow_states(body_state, else_state) + + def visit_IfExp(self, node: ast.IfExp) -> None: + self.visit(node.test) + before = self._flow_state() + self.visit(node.body) + body_state = self._flow_state() + self._restore_flow_state(before) + self.visit(node.orelse) + else_state = self._flow_state() + self._merge_flow_states(body_state, else_state) + def visit_For(self, node: ast.For) -> None: self.visit(node.iter) + before = self._flow_state() + values = _static_string_list(node.iter, self.constants) self._bind_iteration_target( node.target, - _static_string_list(node.iter, self.constants), + values, ) - for statement in (*node.body, *node.orelse): + for statement in node.body: + self.visit(statement) + body_state = self._flow_state() + if values == (): + self._restore_flow_state(before) + elif values is None: + self._merge_flow_states(before, body_state) + for statement in node.orelse: self.visit(statement) def visit_AsyncFor(self, node: ast.AsyncFor) -> None: @@ -2093,6 +2253,65 @@ def f(df, col): ) +def test_conditional_and_loop_flow_joins_never_restore_stale_constants() -> None: + """Branch disagreement and optional loop execution become opaque.""" + + conditional_sources = ( + """ +def f(df, flag): + column = "person_support_channel" + if flag: + column = "age" + return df[column] +""", + """ +def f(df, flag): + if flag: + column = "person_support_channel" + else: + column = "age" + return df[column] +""", + """ +def f(df, flag): + query = df.query + if flag: + query = print + return query("person_support_channel == 1") +""", + """ +def f(df, columns): + column = "age" + for column in columns: + pass + return df[column] +""", + ) + empty_loop_guarded = """ +def f(df): + column = "person_support_channel" + for column in (): + column = "age" + return df[column] +""" + empty_loop_benign = empty_loop_guarded.replace( + '"person_support_channel"', + '"age"', + ).replace( + 'column = "age"\n return', + 'column = "person_support_channel"\n return', + ) + + for source in conditional_sources: + accesses = _source_spine_accesses(source) + assert accesses, source + assert any("fail-closed" in item for item in accesses) + guarded_accesses = _source_spine_accesses(empty_loop_guarded) + assert guarded_accesses + assert any("person_support_channel" in item for item in guarded_accesses) + assert _source_spine_accesses(empty_loop_benign) == () + + def test_subscript_selectors_resolve_or_fail_closed() -> None: """Every round-4 subscript evasion is named or explicitly opaque.""" From 7fa05a40ec16598cd1c7fa11b5453bc13ecc970a Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 00:44:02 -0400 Subject: [PATCH 48/94] Record validation and format the completed guard --- PROGRESS.md | 27 +++++++++++++--- .../tests/test_us_spine_blindness.py | 32 ++++++++----------- 2 files changed, 36 insertions(+), 23 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 4926f824..e5fac6c7 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -19,6 +19,19 @@ Static iterable propagation now covers all built-in literal forms requested by the contract. Conditional statements and optional loop execution now merge bindings conservatively instead of retaining the last syntactically visited value. +All explicit rounds 2-4 reviewer repros and the expanded adversarial +composition battery are green. Full package validation and repository-wide +Ruff pass. + +One literal-contract blocker remains: the receiver inference deliberately does +not classify annotated `pd.DataFrame` parameters or `Frame.table(...)` results +as subscript containers. Enabling only annotated DataFrame inference exposes +about 150 existing dynamic accesses across roughly 39 modules in the pinned +graph; adding table-result inference exposes more. Therefore the current +54-module clean result still has a typed-container third state and cannot +truthfully certify the requested universal “every column surface” wording +without either broad runtime rewrites/interprocedural proof or an explicitly +narrower contract. No push or external mutation is authorized. @@ -122,14 +135,18 @@ No push or external mutation is authorized. - Preserved exact pre-loop bindings across provably empty static iterables. - Added guarded, benign, conditional-alias, optional-loop, and empty-loop binding tests. The guard file now passes 73 tests with the graph clean. +- Ran the full `populace-build` suite: 3,295 passed, 85 skipped, 5 warnings. +- Ran repository-wide `ruff check .`: passed. +- Ran repository-wide `ruff format --check .`: found the existing 44-file + formatting baseline; formatted only this guard file and replayed its 73 tests. ## Next -- Commit flow-sensitive branch and loop joins. -- Reconcile the universal column-container contract with the typed runtime - graph findings from the independent audit. -- Run the guard file, the full `populace-build` suite, Ruff, and the 54-module - graph scan. +- Resolve the typed DataFrame/`Frame.table(...)` third state. The literal + contract requires broad static resolution or runtime-module rewrites; merely + suppressing those findings would violate the “never allowlisted” order. +- After that resolution, rerun the guard, full `populace-build` suite, and + repository-wide Ruff before declaring the lane complete. - Write `/private/tmp/583_fix2_handoff.md` with per-subsystem rules, binding tests, any operator rewrite rationale, exact validation results, and commit inventory. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index dea6c671..f2ab846b 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -492,12 +492,8 @@ def _static_literal_value( for key in node.keys if key is not None ) - values = tuple( - _static_literal_value(value, constants) for value in node.values - ) - if any( - value is _OPAQUE_STATIC_VALUE for value in (*keys, *values) - ): + values = tuple(_static_literal_value(value, constants) for value in node.values) + if any(value is _OPAQUE_STATIC_VALUE for value in (*keys, *values)): return _OPAQUE_STATIC_VALUE try: return dict(zip(keys, values, strict=True)) @@ -533,9 +529,7 @@ def _resolve_static_format( keywords[keyword.arg] = _static_format_value(keyword.value, constants) continue expanded = _static_literal_value(keyword.value, constants) - if isinstance(expanded, dict) and all( - isinstance(key, str) for key in expanded - ): + if isinstance(expanded, dict) and all(isinstance(key, str) for key in expanded): keywords.update(expanded) else: expanded_keywords = True @@ -1099,9 +1093,7 @@ def _bind_iteration_target( self.column_containers[-1][name] = False self.attribute_containers[-1][name] = False self.method_aliases[-1][name] = ( - _OPAQUE_METHOD_ALIAS - if name in self.method_alias_history[-1] - else None + _OPAQUE_METHOD_ALIAS if name in self.method_alias_history[-1] else None ) if self.scope_kinds[-1] == "comprehension": self.assignment_counts[-1][name] = 1 @@ -1168,7 +1160,14 @@ def _merge_flow_states( set[str], ], ) -> None: - left_bindings, left_constants, left_columns, left_attributes, left_aliases, left_history = left + ( + left_bindings, + left_constants, + left_columns, + left_attributes, + left_aliases, + left_history, + ) = left ( right_bindings, right_constants, @@ -1213,8 +1212,7 @@ def _merge_flow_states( for name in names } self.attribute_containers[-1] = { - name: left_attributes.get(name, False) - or right_attributes.get(name, False) + name: left_attributes.get(name, False) or right_attributes.get(name, False) for name in names } self.method_aliases[-1] = {} @@ -2359,9 +2357,7 @@ def f(df): assert all("person_support_channel" in access for access in walrus_accesses) comprehension_accesses = _source_spine_accesses(comprehension_walrus) assert comprehension_accesses - assert all( - "person_support_channel" in access for access in comprehension_accesses - ) + assert all("person_support_channel" in access for access in comprehension_accesses) for source in (multiplication, percent_format, replace_chain): accesses = _source_spine_accesses(source) From b219b2194fbb8755f3b058300e1f4e2df64d250f Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 00:46:12 -0400 Subject: [PATCH 49/94] Finalize PR 583 handoff journal --- PROGRESS.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index e5fac6c7..8900942f 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -22,6 +22,7 @@ conservatively instead of retaining the last syntactically visited value. All explicit rounds 2-4 reviewer repros and the expanded adversarial composition battery are green. Full package validation and repository-wide Ruff pass. +The requested handoff is written at `/private/tmp/583_fix2_handoff.md`. One literal-contract blocker remains: the receiver inference deliberately does not classify annotated `pd.DataFrame` parameters or `Frame.table(...)` results @@ -139,6 +140,9 @@ No push or external mutation is authorized. - Ran repository-wide `ruff check .`: passed. - Ran repository-wide `ruff format --check .`: found the existing 44-file formatting baseline; formatted only this guard file and replayed its 73 tests. +- Wrote `/private/tmp/583_fix2_handoff.md` with per-subsystem rules and binding + tests, the no-rewrite rationale, exact validation receipts, commit inventory, + and the remaining typed-container blocker. ## Next @@ -147,6 +151,3 @@ No push or external mutation is authorized. suppressing those findings would violate the “never allowlisted” order. - After that resolution, rerun the guard, full `populace-build` suite, and repository-wide Ruff before declaring the lane complete. -- Write `/private/tmp/583_fix2_handoff.md` with per-subsystem rules, binding - tests, any operator rewrite rationale, exact validation results, and commit - inventory. From 03350e6bc066860f0a1a5736b6dbe73c85d82fd9 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 00:49:36 -0400 Subject: [PATCH 50/94] Start PR 583 fix-3 completion journal --- PROGRESS.md | 56 ++++++++++++++++++----------------------------------- 1 file changed, 19 insertions(+), 37 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 8900942f..ef543bb6 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,43 +1,23 @@ -# PR #583 Guard Completion Progress +# PR #583 Fix-3 Guard Completion Progress ## State -Round-4 hold remediation is in progress on `multispine-pool-build-578`, from -clean starting commit `19a7a1d`. Subscript resolution now catches all six -round-4 evasions; loop and comprehension targets now propagate every static -string choice or an opaque shadow; and the 54-module operator graph remains -clean. `str.format` now resolves all requested static field forms precisely. -Direct and aliased strict pandas methods now share the same checks, and dynamic -DataFrame `getattr` is fail-closed. Closure free variables now obey lexical -late-binding assignment counts. The five requested subsystems, consolidated -rounds 2-4 completeness invariant, benign battery, exact graph cardinality, -and completed-contract docstrings are implemented. Adversarial audit and full -validation remain. Adversarial alias composition probes are now closed. -Starred and nested format-field composition is now closed too, as are -late-bound writes through comprehensions, `nonlocal`, and `global`. -Static iterable propagation now covers all built-in literal forms requested by -the contract. -Conditional statements and optional loop execution now merge bindings -conservatively instead of retaining the last syntactically visited value. -All explicit rounds 2-4 reviewer repros and the expanded adversarial -composition battery are green. Full package validation and repository-wide -Ruff pass. -The requested handoff is written at `/private/tmp/583_fix2_handoff.md`. - -One literal-contract blocker remains: the receiver inference deliberately does -not classify annotated `pd.DataFrame` parameters or `Frame.table(...)` results -as subscript containers. Enabling only annotated DataFrame inference exposes -about 150 existing dynamic accesses across roughly 39 modules in the pinned -graph; adding table-result inference exposes more. Therefore the current -54-module clean result still has a typed-container third state and cannot -truthfully certify the requested universal “every column surface” wording -without either broad runtime rewrites/interprocedural proof or an explicitly -narrower contract. +Fix-3 completion is in progress on `multispine-pool-build-578`, from clean +starting commit `b219b21`. The populace owner resolved the typed-DataFrame +decision: parameterized `df[column]` is a permitted boundary, while every +statically resolvable guarded column name is contraband in every expression +position in a non-owner operator module. The completed contract must document +that composition argument and the sole accepted residual of names materialized +purely from runtime file/config/environment data. Guard implementation, binding +tests, a clean 54-module scan, full validation, and the fix-3 handoff remain. No push or external mutation is authorized. ## Done +- Read `CLAUDE.md` and `/private/tmp/583_fix2_handoff.md`. +- Confirmed the required branch, clean worktree, and starting HEAD `b219b21`. +- Recorded the owner's explicit typed-parameter and contraband-literal decision. - Read `CLAUDE.md` and the round-4 review log. - Confirmed the required clean starting HEAD and branch. - Confirmed the local GitNexus index is absent; direct AST/source tracing will @@ -146,8 +126,10 @@ No push or external mutation is authorized. ## Next -- Resolve the typed DataFrame/`Frame.table(...)` third state. The literal - contract requires broad static resolution or runtime-module rewrites; merely - suppressing those findings would violate the “never allowlisted” order. -- After that resolution, rerun the guard, full `populace-build` suite, and - repository-wide Ruff before declaring the lane complete. +- Implement the all-expression contraband-literal rule and completed-contract + docstring, keeping annotations and docstrings exempt. +- Add the requested violation/pass/owner tests and adjudicate any findings in + the pinned 54-module graph with the smallest owner-side change. +- Run the guard file, full `populace-build` suite, and repository-wide Ruff. +- Write `/private/tmp/583_fix3_handoff.md`, then restore `PROGRESS.md` to + `origin/main` and commit that restoration so root journals stay out of the PR. From ddb40d5922cdc9d329c31f3129ac706e43c30c0f Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 01:01:47 -0400 Subject: [PATCH 51/94] Close spine guard with contraband literals --- PROGRESS.md | 31 +- .../populace/build/us_runtime/adult_care.py | 6 +- .../populace/build/us_runtime/ssi_take_up.py | 3 +- .../build/us_runtime/support_provenance.py | 2 + .../tests/test_us_spine_blindness.py | 406 ++++++++++++++++-- 5 files changed, 401 insertions(+), 47 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index ef543bb6..932ef473 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -6,10 +6,11 @@ Fix-3 completion is in progress on `multispine-pool-build-578`, from clean starting commit `b219b21`. The populace owner resolved the typed-DataFrame decision: parameterized `df[column]` is a permitted boundary, while every statically resolvable guarded column name is contraband in every expression -position in a non-owner operator module. The completed contract must document -that composition argument and the sole accepted residual of names materialized -purely from runtime file/config/environment data. Guard implementation, binding -tests, a clean 54-module scan, full validation, and the fix-3 handoff remain. +position in a non-owner operator module. The three-layer contract, composition +argument, sole runtime-data residual, and annotation/docstring exemptions are +now documented and executable. All requested binding tests pass, and both the +all-runtime scan and pinned 54-module graph are clean after three narrow +adjudications. Full package validation and the fix-3 handoff remain. No push or external mutation is authorized. @@ -18,6 +19,22 @@ No push or external mutation is authorized. - Read `CLAUDE.md` and `/private/tmp/583_fix2_handoff.md`. - Confirmed the required branch, clean worktree, and starting HEAD `b219b21`. - Recorded the owner's explicit typed-parameter and contraband-literal decision. +- Added the all-expression contraband rule on top of the existing flow-sensitive + string resolver, including static f-string alternatives from enumerated + comprehension choices. +- Exempted only true module/class/function docstrings and annotation syntax; + executable defaults, including lambda defaults, remain checked. +- Added binding tests for the typed parameter boundary, guarded and benign + helper calls, list/tuple/set/dict positions, comparisons, returns, + assignments, defaults, every requested composition form, static f-string + enumeration, owners, annotations, and docstrings. +- Classified `operator_boundary.py` as a reviewed provenance owner because it + enumerates provenance columns only to reject preassembled source frames. +- Moved the shared `person_support_channel` spelling into the existing + `support_provenance.py` owner and imported it into `adult_care.py` and + `ssi_take_up.py`, leaving both treatment modules fully guarded. +- Ran the guard file: 94 passed, with the all-runtime and exact 54-module graph + scans clean. Focused Ruff and `git diff --check` pass. - Read `CLAUDE.md` and the round-4 review log. - Confirmed the required clean starting HEAD and branch. - Confirmed the local GitNexus index is absent; direct AST/source tracing will @@ -126,10 +143,6 @@ No push or external mutation is authorized. ## Next -- Implement the all-expression contraband-literal rule and completed-contract - docstring, keeping annotations and docstrings exempt. -- Add the requested violation/pass/owner tests and adjudicate any findings in - the pinned 54-module graph with the smallest owner-side change. -- Run the guard file, full `populace-build` suite, and repository-wide Ruff. +- Run the full `populace-build` suite and repository-wide Ruff. - Write `/private/tmp/583_fix3_handoff.md`, then restore `PROGRESS.md` to `origin/main` and commit that restoration so root journals stay out of the PR. diff --git a/packages/populace-build/src/populace/build/us_runtime/adult_care.py b/packages/populace-build/src/populace/build/us_runtime/adult_care.py index 92837817..2eb4bcba 100644 --- a/packages/populace-build/src/populace/build/us_runtime/adult_care.py +++ b/packages/populace-build/src/populace/build/us_runtime/adult_care.py @@ -72,6 +72,7 @@ ) from populace.build.us_runtime.support_provenance import ( BASE_ASEC_SUPPORT_CHANNEL, + PERSON_SUPPORT_CHANNEL_COLUMN, has_support_role_metadata, support_role_series, ) @@ -104,7 +105,7 @@ "PEDISDRS", "spm_unit_pre_subsidy_childcare_expenses", "is_full_time_college_student", - "person_support_channel", + PERSON_SUPPORT_CHANNEL_COLUMN, ) # 26 USC 21(b)(1)(A): a dependent under this age qualifies by age (the # engine's gov.irs.credits.cdcc.eligibility.child_age parameter carries the @@ -124,7 +125,6 @@ _PERSON_WEIGHT_COLUMN = "person_weight" _TAX_UNIT_WEIGHT_COLUMN = "adult_care_tax_unit_weight" _SPM_UNIT_WEIGHT_COLUMN = "adult_care_spm_unit_weight" -_PERSON_SUPPORT_CHANNEL_COLUMN = "person_support_channel" _ROLE_COLUMN = "tax_unit_role_input" _AGE_COLUMN = "age" _FLAG_SHARE_BAND = (0.002, 0.12) @@ -262,7 +262,7 @@ def derive_us_adult_care_from_manifest( "self_care_difficulty_source": _SELF_CARE_SOURCE, "childcare_expense_source": _CHILDCARE_SOURCE, "full_time_student_source": _STUDENT_SOURCE, - "support_channel_source": _PERSON_SUPPORT_CHANNEL_COLUMN, + "support_channel_source": PERSON_SUPPORT_CHANNEL_COLUMN, "child_qualifying_age_limit": US_ADULT_CARE_CHILD_QUALIFYING_AGE_LIMIT, "earned_income_sources": list(US_ADULT_CARE_EARNED_INCOME_SOURCES), "seed_from_build_config": True, diff --git a/packages/populace-build/src/populace/build/us_runtime/ssi_take_up.py b/packages/populace-build/src/populace/build/us_runtime/ssi_take_up.py index ff5a4773..abd761ca 100644 --- a/packages/populace-build/src/populace/build/us_runtime/ssi_take_up.py +++ b/packages/populace-build/src/populace/build/us_runtime/ssi_take_up.py @@ -73,6 +73,7 @@ from populace.build.source_manifest import SourceStageSpec, load_source_manifest from populace.build.us_runtime.support_provenance import ( BASE_ASEC_SUPPORT_CHANNEL, + PERSON_SUPPORT_CHANNEL_COLUMN, PUF_TAX_DETAIL_SUPPORT_CHANNEL, has_support_role_metadata, support_clone_index_column, @@ -140,7 +141,7 @@ "age", US_SSI_TAKE_UP_ANCHOR, "person_source_id", - "person_support_channel", + PERSON_SUPPORT_CHANNEL_COLUMN, ) _OUTPUT = US_SSI_TAKE_UP_OUTPUT_COLUMNS[0] diff --git a/packages/populace-build/src/populace/build/us_runtime/support_provenance.py b/packages/populace-build/src/populace/build/us_runtime/support_provenance.py index d5d68067..62381818 100644 --- a/packages/populace-build/src/populace/build/us_runtime/support_provenance.py +++ b/packages/populace-build/src/populace/build/us_runtime/support_provenance.py @@ -10,6 +10,7 @@ __all__ = [ "BASE_ASEC_SUPPORT_CHANNEL", + "PERSON_SUPPORT_CHANNEL_COLUMN", "PUF_TAX_DETAIL_CLONE_INDEX", "PUF_TAX_DETAIL_SUPPORT_CHANNEL", "SPINE_ASSEMBLY_MANIFEST_KEY", @@ -28,6 +29,7 @@ ] BASE_ASEC_SUPPORT_CHANNEL = "asec" +PERSON_SUPPORT_CHANNEL_COLUMN = "person_support_channel" PUF_TAX_DETAIL_SUPPORT_CHANNEL = "puf_tax_detail" PUF_TAX_DETAIL_CLONE_INDEX = 1 SPINE_ASSEMBLY_MANIFEST_KEY = "us_spine_assembly_manifest" diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index f2ab846b..30a83a87 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -1,12 +1,26 @@ -"""Fail-closed guard for source-spine-blind US population operators. - -In every non-owner operator module, each column-access surface must either -resolve to static string choices that are checked by name or be recorded as a -fail-closed violation; there is no silent third state. Covered surfaces are -subscript and ``.loc`` reads/writes, attributes and dynamic ``getattr``, -canonical column factories, and direct or aliased pandas ``get``, ``filter``, -``query``, and ``eval`` calls. Loop/comprehension choices and closure free -variables obey the same contract. +"""Completed fail-closed guard for source-spine-blind US operators. + +The contract has three layers in every non-owner operator module: + +1. Every column-access surface resolves to static choices that are checked by + name or fails closed: subscripts and ``.loc`` reads/writes, attributes, + ``getattr``, canonical column factories, and direct or aliased pandas + ``get``, ``filter``, ``query``, and ``eval`` calls. +2. Opacity is a violation at every strict surface; unresolved selectors, + expressions, aliases, and expanded arguments never create a silent state. +3. Guarded column names are contraband anywhere statically visible in an + executable expression in a non-owner module, including calls, containers, + comparisons, returns, defaults, and assignments. + +A dynamic ``helper(df, column)`` can reach a guarded column only through its +name, and that name must be written somewhere. Wherever it is statically +written, layer 3 catches it using the same constant and string-composition +resolution as the access checks. Therefore a typed parameter-to-parameter +``df[column]`` subscript is deliberately permitted: its producing site bears +the name check. The single accepted residual is a guarded name materialized +purely from runtime file, configuration, or environment content, where no +static spelling exists to inspect. Annotations and docstrings are exempt +because they are not dataflow. """ from __future__ import annotations @@ -46,6 +60,8 @@ _SOURCE_SPINE_PROVENANCE_OWNERS = frozenset( { "base_pool.py", # Legacy late-spine assembly. + # Enumerates provenance columns only to reject preassembled source frames. + "operator_boundary.py", "puf_qrf_chain.py", # Carries provenance into resumable checkpoints. "puf_support.py", # Validates provenance at the clone boundary. "spine_agreement.py", # Pre-calibration distribution comparison. @@ -699,12 +715,94 @@ def _static_string_values( node: ast.AST, constants: list[dict[str, object]], ) -> tuple[str, ...] | None: + if isinstance(node, ast.JoinedStr): + alternatives = _static_joined_string_values(node, constants) + if alternatives is not None: + return alternatives shape = _static_string_shape(node, constants) if shape is not None: return (shape,) return _static_string_list(node, constants) +def _static_joined_string_values( + node: ast.JoinedStr, + constants: list[dict[str, object]], +) -> tuple[str, ...] | None: + """Expand every statically enumerable f-string field choice.""" + + assembled = ("",) + for part in node.values: + if isinstance(part, ast.Constant) and isinstance(part.value, str): + choices = (part.value,) + elif isinstance(part, ast.FormattedValue): + choices = _static_string_values(part.value, constants) + if choices is None: + return None + format_spec = ( + "" + if part.format_spec is None + else _static_string_shape(part.format_spec, constants) + ) + if format_spec is None or _OPAQUE_STRING_PART in format_spec: + return None + formatted: list[str] = [] + for choice in choices: + if part.conversion == ord("s"): + converted = str(choice) + elif part.conversion == ord("r"): + converted = repr(choice) + elif part.conversion == ord("a"): + converted = ascii(choice) + else: + converted = choice + try: + formatted.append(format(converted, format_spec)) + except (TypeError, ValueError): + return None + choices = tuple(formatted) + else: + choices = _static_string_values(part, constants) + if choices is None: + return None + assembled = tuple(prefix + choice for prefix in assembled for choice in choices) + return assembled + + +def _statically_visible_source_columns( + node: ast.AST, + constants: list[dict[str, object]], +) -> tuple[str, ...]: + """Return guarded names present in a statically resolvable expression.""" + + values = _static_string_values(node, constants) + if values is None: + return () + return tuple( + column + for column in sorted(_OPERATOR_SOURCE_COLUMNS) + if any(column in value for value in values) + ) + + +def _docstring_value_ids(tree: ast.AST) -> set[int]: + """Identify only true scope docstrings, not arbitrary string expressions.""" + + scope_nodes = (ast.Module, ast.ClassDef, ast.FunctionDef, ast.AsyncFunctionDef) + docstrings: set[int] = set() + for scope in ast.walk(tree): + if not isinstance(scope, scope_nodes) or not scope.body: + continue + statement = scope.body[0] + if ( + isinstance(statement, ast.Expr) + and isinstance(statement.value, ast.Constant) + and isinstance(statement.value.value, str) + ): + docstrings.add(id(statement.value)) + return docstrings + + def _pandas_expression_source(node: ast.AST) -> str | None: """Resolve guarded column-shaped tokens in a static pandas expression.""" @@ -889,8 +987,14 @@ def _scope_assignment_counts( class _SourceReadVisitor(ast.NodeVisitor): - def __init__(self, factory_aliases: set[str]) -> None: + def __init__( + self, + factory_aliases: set[str], + *, + docstring_value_ids: set[int], + ) -> None: self.factory_aliases = factory_aliases + self.docstring_value_ids = docstring_value_ids self.bindings: list[dict[str, str | None]] = [{}] self.constants: list[dict[str, object]] = [{}] self.column_containers: list[dict[str, bool]] = [{}] @@ -901,6 +1005,15 @@ def __init__(self, factory_aliases: set[str]) -> None: self.scope_kinds = ["module"] self.accesses: set[tuple[int, int, str]] = set() + def visit(self, node: ast.AST) -> object: + if isinstance(node, ast.expr) and id(node) not in self.docstring_value_ids: + for column in _statically_visible_source_columns(node, self.constants): + self._record( + node, + f"contraband source column {column!r}", + ) + return super().visit(node) + def _expression(self, node: ast.AST) -> str | None: return _source_expression( node, @@ -1451,6 +1564,9 @@ def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None: self._visit_function(node) def visit_Lambda(self, node: ast.Lambda) -> None: + for default in (*node.args.defaults, *node.args.kw_defaults): + if default is not None: + self.visit(default) arguments = ( *node.args.posonlyargs, *node.args.args, @@ -1773,11 +1889,14 @@ def visit_Subscript(self, node: ast.Subscript) -> None: def _source_spine_accesses(source: str) -> tuple[str, ...]: - """Describe named or fail-closed violations on every column surface.""" + """Describe strict-surface and static-name contraband violations.""" tree = ast.parse(source) aliases = _factory_aliases(tree) - visitor = _SourceReadVisitor(aliases) + visitor = _SourceReadVisitor( + aliases, + docstring_value_ids=_docstring_value_ids(tree), + ) visitor.visit(tree) return tuple( f"line {line}:{column + 1}: {description}" @@ -1785,6 +1904,17 @@ def _source_spine_accesses(source: str) -> tuple[str, ...]: ) +def _non_owner_source_spine_accesses( + module_name: str, + source: str, +) -> tuple[str, ...]: + """Apply the guard unless the module is a reviewed provenance owner.""" + + if module_name in _SOURCE_SPINE_PROVENANCE_OWNERS: + return () + return _source_spine_accesses(source) + + def _called_function_names(source: str) -> set[str]: tree = ast.parse(source) return { @@ -1932,13 +2062,14 @@ def test_us_runtime_frame_rebuilds_preserve_immutable_metadata() -> None: def test_runtime_population_operators_are_source_spine_blind() -> None: - """Every operator column surface resolves-and-checks or fails closed. + """Every operator obeys the strict-surface and contraband-name contract. The guard parses executable syntax rather than searching raw text, so - comments, docstrings, and source-manifest declarations may explain the - invariant without creating an exception. It covers subscript/``.loc`` - reads and writes, attributes/``getattr``, canonical factories, and direct - or aliased ``get``, ``filter``, ``query``, and ``eval`` calls. + comments, annotations, and docstrings may explain the invariant. Executable + declarations are checked like every other expression. Strict surfaces + include subscript/``.loc`` reads and writes, attributes/``getattr``, + canonical factories, and direct or aliased ``get``, ``filter``, ``query``, + and ``eval`` calls. """ missing_owners = sorted( @@ -1953,9 +2084,7 @@ def test_runtime_population_operators_are_source_spine_blind() -> None: offenders: dict[str, tuple[str, ...]] = {} for path in sorted(_US_RUNTIME.glob("*.py")): - if path.name in _SOURCE_SPINE_PROVENANCE_OWNERS: - continue - accesses = _source_spine_accesses(path.read_text()) + accesses = _non_owner_source_spine_accesses(path.name, path.read_text()) if accesses: offenders[path.name] = accesses @@ -2020,9 +2149,7 @@ def test_pool_build_tool_import_graph_is_source_spine_blind() -> None: offenders: dict[str, tuple[str, ...]] = {} for path in (tool, *runtime_graph): - if path.name in _SOURCE_SPINE_PROVENANCE_OWNERS: - continue - reads = _source_spine_accesses(path.read_text()) + reads = _non_owner_source_spine_accesses(path.name, path.read_text()) if reads: offenders[str(path.relative_to(_REPOSITORY_ROOT))] = reads assert not offenders, ( @@ -2154,6 +2281,215 @@ def op(df): assert _source_spine_accesses(benign_reads) == () +def test_typed_parameter_subscript_is_closed_at_static_call_sites() -> None: + """Typed helper subscripts are permitted; their static producers are not.""" + + guarded = """ +def select(df: pd.DataFrame, column: str): + return df[column] +def caller(df: pd.DataFrame): + return select(df, "person_support_channel") +""" + benign = guarded.replace('"person_support_channel"', '"age"') + typed_boundary = """ +def select(df: pd.DataFrame, column: str): + return df[column] +""" + + accesses = _source_spine_accesses(guarded) + assert len(accesses) == 1 + assert accesses[0].startswith("line 5:") + assert "contraband source column 'person_support_channel'" in accesses[0] + assert "fail-closed" not in accesses[0] + assert _source_spine_accesses(benign) == () + assert _source_spine_accesses(typed_boundary) == () + + +@pytest.mark.parametrize( + ("position", "source"), + ( + ( + "call argument", + """ +def f(df): + return helper(df, "person_support_channel") +""", + ), + ( + "list element", + """ +VALUE = ["person_support_channel"] +""", + ), + ( + "tuple element", + """ +VALUE = ("person_support_channel",) +""", + ), + ( + "set element", + """ +VALUE = {"person_support_channel"} +""", + ), + ( + "dict key", + """ +VALUE = {"person_support_channel": "source"} +""", + ), + ( + "dict value", + """ +VALUE = {"column": "person_support_channel"} +""", + ), + ( + "comparison", + """ +def f(value): + return value == "person_support_channel" +""", + ), + ( + "return value", + """ +def f(): + return "person_support_channel" +""", + ), + ( + "assignment", + """ +VALUE = "person_support_channel" +""", + ), + ( + "function default", + """ +def f(column="person_support_channel"): + return column +""", + ), + ( + "lambda default", + """ +f = lambda column="person_support_channel": column +""", + ), + ), +) +def test_contraband_names_are_rejected_in_every_expression_position( + position: str, + source: str, +) -> None: + """Every executable static spelling is a named, non-opaque violation.""" + + accesses = _source_spine_accesses(source) + assert accesses, position + assert all("person_support_channel" in access for access in accesses) + assert all("fail-closed" not in access for access in accesses) + + +@pytest.mark.parametrize( + ("composition", "expression"), + ( + ("concatenation", '"person_support" + "_channel"'), + ("format", '"{}_support_channel".format("person")'), + ("percent", '"%s_support_channel" % "person"'), + ("replace", '"person_x".replace("x", "support_channel")'), + ("multiplication", '"person_support_channel" * 1'), + ), +) +def test_contraband_names_reuse_static_expression_resolution( + composition: str, + expression: str, +) -> None: + """Composition is resolved outside known pandas access surfaces too.""" + + source = f"def f():\n return sink({expression})\n" + accesses = _source_spine_accesses(source) + assert accesses, composition + assert all("person_support_channel" in access for access in accesses) + assert all("fail-closed" not in access for access in accesses) + + +def test_contraband_names_resolve_bound_and_enumerated_fragments() -> None: + """Bound concatenation and static f-string choices expose exact names.""" + + bound = """ +PREFIX = "person_support" +def f(): + return sink(PREFIX + "_channel") +""" + enumerated = """ +ENTITIES = ("person", "household") +VALUES = { + f"{entity}_support_channel" + for entity in ENTITIES +} +""" + + bound_accesses = _source_spine_accesses(bound) + assert any("person_support_channel" in access for access in bound_accesses) + enumerated_accesses = _source_spine_accesses(enumerated) + assert any("person_support_channel" in access for access in enumerated_accesses) + assert any("household_support_channel" in access for access in enumerated_accesses) + + +def test_benign_helpers_and_containers_remain_clean() -> None: + """Ordinary dynamic helpers and statically benign payloads are permitted.""" + + source = """ +def select(df: pd.DataFrame, column: str): + return df[column] +def f(df: pd.DataFrame, column="age"): + payload = [ + "age", + ("income",), + {"tenure"}, + {"column": "wages"}, + ] + return select(df, column), payload +""" + assert _source_spine_accesses(source) == () + + +def test_annotations_and_true_docstrings_are_not_dataflow() -> None: + """Every annotation form and each scope's real docstring are exempt.""" + + source = '''"""person_support_channel""" +class Spec: + """person_support_channel""" + field: Literal["person_support_channel"] +def f( + df: pd.DataFrame, + column: Literal["person_support_channel"], +) -> Literal["person_support_channel"]: + """person_support_channel""" + local: Literal["person_support_channel"] + return df[column] +''' + later_expression = """ +def f(): + pass + "person_support_channel" +""" + + assert _source_spine_accesses(source) == () + assert _source_spine_accesses(later_expression) + + +def test_reviewed_provenance_owners_are_unaffected_by_contraband_rule() -> None: + """The same executable spelling is exempt only in a reviewed owner.""" + + source = 'COLUMN = "person_support_channel"\n' + assert _source_spine_accesses(source) + assert _non_owner_source_spine_accesses("support_provenance.py", source) == () + assert _non_owner_source_spine_accesses("future_operator.py", source) + + def test_guard_sees_through_static_indirection_and_fails_closed_on_opacity(): """Sol #583 round-2 bypasses: one level of static indirection must resolve, and anything still opaque is a violation by default.""" @@ -2292,13 +2628,13 @@ def f(df): column = "age" return df[column] """ - empty_loop_benign = empty_loop_guarded.replace( - '"person_support_channel"', - '"age"', - ).replace( - 'column = "age"\n return', - 'column = "person_support_channel"\n return', - ) + empty_loop_benign = """ +def f(df): + column = "age" + for column in (): + column = "income" + return df[column] +""" for source in conditional_sources: accesses = _source_spine_accesses(source) @@ -2353,8 +2689,11 @@ def f(df): """ walrus_accesses = _source_spine_accesses(walrus) - assert len(walrus_accesses) == 2 - assert all("person_support_channel" in access for access in walrus_accesses) + walrus_subscripts = [ + access for access in walrus_accesses if "subscript using" in access + ] + assert len(walrus_subscripts) == 2 + assert all("person_support_channel" in access for access in walrus_subscripts) comprehension_accesses = _source_spine_accesses(comprehension_walrus) assert comprehension_accesses assert all("person_support_channel" in access for access in comprehension_accesses) @@ -2429,7 +2768,6 @@ def f(df, columns): assert _source_spine_accesses(benign) == () guarded_accesses = _source_spine_accesses(guarded) - assert len(guarded_accesses) == 2 assert any("person_support_channel" in item for item in guarded_accesses) assert any("household_spine" in item for item in guarded_accesses) dynamic_accesses = _source_spine_accesses(dynamic) @@ -2929,7 +3267,7 @@ def inner(): return inner() """ later_local_shadow = """ -expr = "person_support_channel == 1" +expr = "income >= 0" def outer(df): def inner(): return df.query(expr) From 4d78471129207a5e0635ecb3da30f0fe2a068889 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 01:19:47 -0400 Subject: [PATCH 52/94] Harden universal contraband traversal --- PROGRESS.md | 17 + .../tests/test_us_spine_blindness.py | 372 ++++++++++++++++-- 2 files changed, 355 insertions(+), 34 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 932ef473..6ec242c5 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -35,6 +35,23 @@ No push or external mutation is authorized. `ssi_take_up.py`, leaving both treatment modules fully guarded. - Ran the guard file: 94 passed, with the all-runtime and exact 54-module graph scans clean. Focused Ruff and `git diff --check` pass. +- Closed post-implementation adversarial findings: function defaults and + decorators now resolve at definition time while bodies retain late-binding + analysis; class namespaces no longer corrupt enclosing constants; PEP 695 + class type parameters remain annotation-exempt; and attribute assignment, + annotated-assignment, loop, and comprehension targets are visited. +- Documented the full non-strict receiver composition boundary for typed, + constructed-DataFrame, and `Frame.table(...)` result subscripts, with no + interprocedural proof and the same sole runtime-data residual. +- Added binding regressions for definition-time constant state, class namespace + isolation, PEP 695 annotations, every attribute-store form, and the broader + truthful subscript boundary. +- Preserved real nested lambda/comprehension closures while excluding class + namespaces, and distinguished abstract static iteration choices so composed + fragments resolve across those closures without treating ordinary tuple + formatting as alternative values. +- The expanded guard file passes 99 tests; the all-runtime and exact 54-module + scans remain clean. - Read `CLAUDE.md` and the round-4 review log. - Confirmed the required clean starting HEAD and branch. - Confirmed the local GitNexus index is absent; direct AST/source tracing will diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 30a83a87..94eb324a 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -2,10 +2,11 @@ The contract has three layers in every non-owner operator module: -1. Every column-access surface resolves to static choices that are checked by - name or fails closed: subscripts and ``.loc`` reads/writes, attributes, - ``getattr``, canonical column factories, and direct or aliased pandas - ``get``, ``filter``, ``query``, and ``eval`` calls. +1. Every column-access surface kind is visited, and every locally strict + occurrence resolves to static choices that are checked by name or fails + closed: subscripts and ``.loc`` reads/writes, attributes, ``getattr``, + canonical column factories, and direct or aliased pandas ``get``, + ``filter``, ``query``, and ``eval`` calls. 2. Opacity is a violation at every strict surface; unresolved selectors, expressions, aliases, and expanded arguments never create a silent state. 3. Guarded column names are contraband anywhere statically visible in an @@ -17,10 +18,12 @@ written, layer 3 catches it using the same constant and string-composition resolution as the access checks. Therefore a typed parameter-to-parameter ``df[column]`` subscript is deliberately permitted: its producing site bears -the name check. The single accepted residual is a guarded name materialized -purely from runtime file, configuration, or environment content, where no -static spelling exists to inspect. Annotations and docstrings are exempt -because they are not dataflow. +the name check. Constructed-DataFrame and ``Frame.table(...)`` result +subscripts outside local strict-receiver inference use the same composition +boundary. No interprocedural proof is attempted. The single accepted residual +is a guarded name materialized purely from runtime file, configuration, or +environment content, where no static spelling exists to inspect. Annotations +and docstrings are exempt because they are not dataflow. """ from __future__ import annotations @@ -458,6 +461,10 @@ def _static_string_shape( _OPAQUE_METHOD_ALIAS = ("", True) +class _StaticStringChoices(tuple): + """Abstract alternatives bound by one static loop/comprehension target.""" + + def _static_format_value( node: ast.AST, constants: list[dict[str, object]], @@ -722,9 +729,49 @@ def _static_string_values( shape = _static_string_shape(node, constants) if shape is not None: return (shape,) + if isinstance(node, ast.BinOp) and isinstance(node.op, ast.Add): + alternatives = _static_concatenated_string_values(node, constants) + if alternatives is not None: + return alternatives return _static_string_list(node, constants) +def _static_composition_operand_values( + node: ast.AST, + constants: list[dict[str, object]], +) -> tuple[str, ...] | None: + """Resolve scalar strings or abstract iteration choices for composition.""" + + if isinstance(node, ast.Name): + for scope in reversed(constants): + if node.id in scope: + value = scope[node.id] + if isinstance(value, _StaticStringChoices): + return tuple(value) + break + if isinstance(node, ast.NamedExpr): + return _static_composition_operand_values(node.value, constants) + shape = _static_string_shape(node, constants) + if shape is not None: + return (shape,) + if isinstance(node, ast.BinOp) and isinstance(node.op, ast.Add): + return _static_concatenated_string_values(node, constants) + return None + + +def _static_concatenated_string_values( + node: ast.BinOp, + constants: list[dict[str, object]], +) -> tuple[str, ...] | None: + """Expand concatenation across abstract loop/comprehension alternatives.""" + + left = _static_composition_operand_values(node.left, constants) + right = _static_composition_operand_values(node.right, constants) + if left is None or right is None: + return None + return tuple(prefix + suffix for prefix in left for suffix in right) + + def _static_joined_string_values( node: ast.JoinedStr, constants: list[dict[str, object]], @@ -736,9 +783,17 @@ def _static_joined_string_values( if isinstance(part, ast.Constant) and isinstance(part.value, str): choices = (part.value,) elif isinstance(part, ast.FormattedValue): - choices = _static_string_values(part.value, constants) - if choices is None: - return None + composition_choices = _static_composition_operand_values( + part.value, + constants, + ) + if composition_choices is not None: + choices: tuple[object, ...] = composition_choices + else: + static_value = _static_format_value(part.value, constants) + if static_value is _OPAQUE_STATIC_VALUE: + return None + choices = (static_value,) format_spec = ( "" if part.format_spec is None @@ -1003,6 +1058,17 @@ def __init__( self.method_alias_history: list[set[str]] = [set()] self.assignment_counts: list[dict[str, int]] = [{}] self.scope_kinds = ["module"] + self.class_lexical_flow_states: list[ + tuple[ + dict[str, str | None], + dict[str, object], + dict[str, bool], + dict[str, bool], + dict[str, tuple[str, bool] | None], + set[str], + ] + ] = [] + self.class_lexical_scope_depths: list[int] = [] self.accesses: set[tuple[int, int, str]] = set() def visit(self, node: ast.AST) -> object: @@ -1157,13 +1223,23 @@ def _bind( for target in targets: self._bind_target(target, value, scope_index=scope_index) + def _visit_access_target(self, target: ast.AST) -> None: + """Visit column-bearing stores without treating names as string uses.""" + + if isinstance(target, (ast.Attribute, ast.Subscript)): + self.visit(target) + elif isinstance(target, ast.Starred): + self._visit_access_target(target.value) + elif isinstance(target, (ast.List, ast.Tuple)): + for element in target.elts: + self._visit_access_target(element) + def visit_Assign(self, node: ast.Assign) -> None: self.visit(node.value) for target in node.targets: - if isinstance(target, ast.Subscript): - self.visit(target) - if isinstance(target.value, ast.Name): - self._poison(target.value.id) + self._visit_access_target(target) + if isinstance(target, ast.Subscript) and isinstance(target.value, ast.Name): + self._poison(target.value.id) self._bind(list(node.targets), node.value) def visit_AugAssign(self, node: ast.AugAssign) -> None: @@ -1174,6 +1250,7 @@ def visit_AugAssign(self, node: ast.AugAssign) -> None: def visit_AnnAssign(self, node: ast.AnnAssign) -> None: if node.value is None: + self._visit_access_target(node.target) for name in _assigned_names(node.target): self.bindings[-1][name] = None self.constants[-1][name] = None @@ -1182,8 +1259,7 @@ def visit_AnnAssign(self, node: ast.AnnAssign) -> None: self.method_aliases[-1][name] = None return self.visit(node.value) - if isinstance(node.target, ast.Subscript): - self.visit(node.target) + self._visit_access_target(node.target) self._bind([node.target], node.value) def visit_NamedExpr(self, node: ast.NamedExpr) -> None: @@ -1202,7 +1278,9 @@ def _bind_iteration_target( ) -> None: for name in _assigned_names(target): self.bindings[-1][name] = None - self.constants[-1][name] = values + self.constants[-1][name] = ( + None if values is None else _StaticStringChoices(values) + ) self.column_containers[-1][name] = False self.attribute_containers[-1][name] = False self.method_aliases[-1][name] = ( @@ -1366,6 +1444,7 @@ def visit_IfExp(self, node: ast.IfExp) -> None: def visit_For(self, node: ast.For) -> None: self.visit(node.iter) + self._visit_access_target(node.target) before = self._flow_state() values = _static_string_list(node.iter, self.constants) self._bind_iteration_target( @@ -1389,6 +1468,16 @@ def _visit_comprehension( self, node: ast.ListComp | ast.SetComp | ast.DictComp | ast.GeneratorExp, ) -> None: + first_generator = node.generators[0] + self.visit(first_generator.iter) + first_values = _static_string_list(first_generator.iter, self.constants) + direct_class_body = bool( + self.class_lexical_scope_depths + and len(self.scope_kinds) == self.class_lexical_scope_depths[-1] + ) + class_body_state = self._flow_state() if direct_class_body else None + if class_body_state is not None: + self._restore_flow_state(self.class_lexical_flow_states[-1]) self.bindings.append({}) self.constants.append({}) self.column_containers.append({}) @@ -1397,11 +1486,15 @@ def _visit_comprehension( self.method_alias_history.append(set()) self.assignment_counts.append({}) self.scope_kinds.append("comprehension") - for generator in node.generators: - self.visit(generator.iter) + for index, generator in enumerate(node.generators): + values = first_values + if index: + self.visit(generator.iter) + values = _static_string_list(generator.iter, self.constants) + self._visit_access_target(generator.target) self._bind_iteration_target( generator.target, - _static_string_list(generator.iter, self.constants), + values, ) for condition in generator.ifs: self.visit(condition) @@ -1418,6 +1511,8 @@ def _visit_comprehension( self.column_containers.pop() self.constants.pop() self.bindings.pop() + if class_body_state is not None: + self._restore_flow_state(class_body_state) def visit_ListComp(self, node: ast.ListComp) -> None: self._visit_comprehension(node) @@ -1434,15 +1529,69 @@ def visit_GeneratorExp(self, node: ast.GeneratorExp) -> None: def visit_TypeAlias(self, node: ast.TypeAlias) -> None: """Type parameters are not executable column selectors.""" + def _visit_function_definition_expressions( + self, + node: ast.FunctionDef | ast.AsyncFunctionDef, + ) -> None: + """Visit expressions evaluated when a function is defined.""" + + for decorator in node.decorator_list: + self.visit(decorator) + for default in (*node.args.defaults, *node.args.kw_defaults): + if default is not None: + self.visit(default) + + def _visit_class_definition( + self, + node: ast.ClassDef, + ) -> list[ast.FunctionDef | ast.AsyncFunctionDef]: + """Visit a class body without leaking its namespace or annotations.""" + + for expression in (*node.decorator_list, *node.bases): + self.visit(expression) + for keyword in node.keywords: + self.visit(keyword.value) + + caller_state = self._flow_state() + lexical_state = ( + self.class_lexical_flow_states[-1] + if self.class_lexical_flow_states + else caller_state + ) + self._restore_flow_state(lexical_state) + self.class_lexical_flow_states.append(lexical_state) + self.class_lexical_scope_depths.append(len(self.scope_kinds)) + deferred: list[ast.FunctionDef | ast.AsyncFunctionDef] = [] + for statement in node.body: + if isinstance(statement, (ast.FunctionDef, ast.AsyncFunctionDef)): + self._visit_function_definition_expressions(statement) + deferred.append(statement) + elif isinstance(statement, ast.ClassDef): + deferred.extend(self._visit_class_definition(statement)) + else: + self.visit(statement) + self.class_lexical_scope_depths.pop() + self.class_lexical_flow_states.pop() + self._restore_flow_state(caller_state) + return deferred + def _visit_scope_statements(self, body: list[ast.stmt]) -> None: deferred: list[ast.FunctionDef | ast.AsyncFunctionDef] = [] for statement in body: if isinstance(statement, (ast.FunctionDef, ast.AsyncFunctionDef)): + self._visit_function_definition_expressions(statement) deferred.append(statement) + elif isinstance(statement, ast.ClassDef): + deferred.extend(self._visit_class_definition(statement)) else: self.visit(statement) for function in deferred: - self.visit(function) + self._visit_function(function, visit_definition_expressions=False) + + def visit_ClassDef(self, node: ast.ClassDef) -> None: + deferred = self._visit_class_definition(node) + for function in deferred: + self._visit_function(function, visit_definition_expressions=False) def visit_Module(self, node: ast.Module) -> None: counts = _scope_assignment_counts( @@ -1476,12 +1625,18 @@ def _unstable_outer_names(self) -> dict[str, bool]: def _visit_function( self, node: ast.FunctionDef | ast.AsyncFunctionDef, + *, + visit_definition_expressions: bool = True, ) -> None: - for decorator in node.decorator_list: - self.visit(decorator) - for default in (*node.args.defaults, *node.args.kw_defaults): - if default is not None: - self.visit(default) + if visit_definition_expressions: + self._visit_function_definition_expressions(node) + direct_class_body = bool( + self.class_lexical_scope_depths + and len(self.scope_kinds) == self.class_lexical_scope_depths[-1] + ) + class_body_state = self._flow_state() if direct_class_body else None + if class_body_state is not None: + self._restore_flow_state(self.class_lexical_flow_states[-1]) arguments = ( *node.args.posonlyargs, *node.args.args, @@ -1556,6 +1711,8 @@ def _visit_function( self.column_containers.pop() self.constants.pop() self.bindings.pop() + if class_body_state is not None: + self._restore_flow_state(class_body_state) def visit_FunctionDef(self, node: ast.FunctionDef) -> None: self._visit_function(node) @@ -1567,6 +1724,13 @@ def visit_Lambda(self, node: ast.Lambda) -> None: for default in (*node.args.defaults, *node.args.kw_defaults): if default is not None: self.visit(default) + direct_class_body = bool( + self.class_lexical_scope_depths + and len(self.scope_kinds) == self.class_lexical_scope_depths[-1] + ) + class_body_state = self._flow_state() if direct_class_body else None + if class_body_state is not None: + self._restore_flow_state(self.class_lexical_flow_states[-1]) arguments = ( *node.args.posonlyargs, *node.args.args, @@ -1630,6 +1794,8 @@ def visit_Lambda(self, node: ast.Lambda) -> None: self.column_containers.pop() self.constants.pop() self.bindings.pop() + if class_body_state is not None: + self._restore_flow_state(class_body_state) _MUTATORS = frozenset( { @@ -2297,14 +2463,121 @@ def select(df: pd.DataFrame, column: str): """ accesses = _source_spine_accesses(guarded) - assert len(accesses) == 1 - assert accesses[0].startswith("line 5:") - assert "contraband source column 'person_support_channel'" in accesses[0] - assert "fail-closed" not in accesses[0] + caller_accesses = [access for access in accesses if access.startswith("line 5:")] + assert caller_accesses + assert all( + "contraband source column 'person_support_channel'" in access + for access in caller_accesses + ) + assert not any(access.startswith(("line 2:", "line 3:")) for access in accesses) assert _source_spine_accesses(benign) == () assert _source_spine_accesses(typed_boundary) == () +def test_non_strict_subscript_forms_share_the_composition_boundary() -> None: + """Receiver forms outside local strict inference rely on the producer rule.""" + + sources = ( + """ +def f(column: str): + df = pd.DataFrame() + return df[column] +""", + """ +def f(frame: Frame, column: str): + return frame.table("person")[column] +""", + """ +def f(df: pd.DataFrame): + return df[load_runtime_column()] +""", + ) + + for source in sources: + assert _source_spine_accesses(source) == (), source + + +def test_function_definition_expressions_use_definition_time_constants() -> None: + """Defaults and decorators resolve before later enclosing rebindings.""" + + guarded_default = """ +PREFIX = "person_support" +def f(column=PREFIX + "_channel"): + return column +PREFIX = "age" +""" + guarded_decorator = """ +PREFIX = "person_support" +@register(PREFIX + "_channel") +def f(): + pass +PREFIX = "age" +""" + nested_guarded_default = """ +def outer(): + prefix = "person_support" + def inner(column=prefix + "_channel"): + return column + prefix = "age" + return inner +""" + benign_default = """ +PREFIX = "age" +def f(column=PREFIX + "_channel"): + return column +PREFIX = "person_support" +""" + + for source in (guarded_default, guarded_decorator, nested_guarded_default): + accesses = _source_spine_accesses(source) + assert accesses, source + assert any("person_support_channel" in access for access in accesses) + assert _source_spine_accesses(benign_default) == () + + +def test_class_namespaces_do_not_overwrite_enclosing_constant_state() -> None: + """Sequential class locals are checked but never leak into outer flow.""" + + guarded_outer = """ +PREFIX = "person_support" +class Spec: + PREFIX = "age" +sink(PREFIX + "_channel") +""" + benign_outer = """ +PREFIX = "age" +class Spec: + PREFIX = "person_support" +sink(PREFIX + "_channel") +""" + + guarded_accesses = _source_spine_accesses(guarded_outer) + assert any("person_support_channel" in access for access in guarded_accesses) + assert _source_spine_accesses(benign_outer) == () + + +def test_class_created_closures_keep_their_real_nonclass_scopes() -> None: + """Leaving class scope must preserve nested lambda/comprehension bindings.""" + + nested_lambda = """ +class Spec: + make = lambda: ( + (prefix := "person_support"), + lambda: prefix + "_channel", + ) +""" + comprehension_lambda = """ +class Spec: + VALUES = ("person_support",) + funcs = [lambda: prefix + "_channel" for prefix in VALUES] +""" + + for source in (nested_lambda, comprehension_lambda): + accesses = _source_spine_accesses(source) + assert accesses, source + assert any("person_support_channel" in access for access in accesses) + + @pytest.mark.parametrize( ("position", "source"), ( @@ -2399,7 +2672,7 @@ def test_contraband_names_are_rejected_in_every_expression_position( ("format", '"{}_support_channel".format("person")'), ("percent", '"%s_support_channel" % "person"'), ("replace", '"person_x".replace("x", "support_channel")'), - ("multiplication", '"person_support_channel" * 1'), + ("multiplication", '"person_" + ("support" * 1) + "_channel"'), ), ) def test_contraband_names_reuse_static_expression_resolution( @@ -2429,6 +2702,10 @@ def f(): f"{entity}_support_channel" for entity in ENTITIES } +""" + ordinary_tuple = """ +ENTITIES = ("person", "age") +VALUE = f"{ENTITIES}_support_channel" """ bound_accesses = _source_spine_accesses(bound) @@ -2436,6 +2713,7 @@ def f(): enumerated_accesses = _source_spine_accesses(enumerated) assert any("person_support_channel" in access for access in enumerated_accesses) assert any("household_support_channel" in access for access in enumerated_accesses) + assert _source_spine_accesses(ordinary_tuple) == () def test_benign_helpers_and_containers_remain_clean() -> None: @@ -2460,10 +2738,14 @@ def test_annotations_and_true_docstrings_are_not_dataflow() -> None: """Every annotation form and each scope's real docstring are exempt.""" source = '''"""person_support_channel""" -class Spec: +type Alias = Literal["person_support_channel"] +class Spec[ + T: Literal["person_support_channel"], + U = Literal["person_support_channel"], +]: """person_support_channel""" field: Literal["person_support_channel"] -def f( +def f[V: Literal["person_support_channel"]]( df: pd.DataFrame, column: Literal["person_support_channel"], ) -> Literal["person_support_channel"]: @@ -2481,6 +2763,28 @@ def f(): assert _source_spine_accesses(later_expression) +def test_attribute_store_surfaces_are_always_visited() -> None: + """Assignments and loop targets cannot hide guarded attribute writes.""" + + source = """ +def f(row, values): + row.person_support_channel = "asec" + row.household_spine_source_id: str = "source" + for row.tax_unit_spine in values: + pass + row.spm_unit_spine += 1 +""" + + accesses = _source_spine_accesses(source) + for column in ( + "person_support_channel", + "household_spine_source_id", + "tax_unit_spine", + "spm_unit_spine", + ): + assert any(column in access for access in accesses), column + + def test_reviewed_provenance_owners_are_unaffected_by_contraband_rule() -> None: """The same executable spelling is exempt only in a reviewed owner.""" From cceb79723540e7bb22ec98417667981a810d650a Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 01:22:24 -0400 Subject: [PATCH 53/94] Record PR 583 fix-3 certification --- PROGRESS.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 6ec242c5..f73dbfbb 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -10,7 +10,8 @@ position in a non-owner operator module. The three-layer contract, composition argument, sole runtime-data residual, and annotation/docstring exemptions are now documented and executable. All requested binding tests pass, and both the all-runtime scan and pinned 54-module graph are clean after three narrow -adjudications. Full package validation and the fix-3 handoff remain. +adjudications. Final guard/package/Ruff certification is green. The fix-3 +handoff and required root-journal restoration remain. No push or external mutation is authorized. @@ -52,6 +53,15 @@ No push or external mutation is authorized. formatting as alternative values. - The expanded guard file passes 99 tests; the all-runtime and exact 54-module scans remain clean. +- Completed independent adversarial review with a final clean verdict after + binding every reported definition-time, class-scope, annotation, store, + closure, and static-format precision repro. +- Ran the final guard file through `uv`: 99 passed. +- Ran the final full `populace-build` suite: 3,321 passed, 85 skipped, 5 known + warnings in 107.66 seconds. +- Ran repository-wide `ruff check .`: passed. +- Ran focused `ruff format --check` on all four changed code/test files: + passed. `git diff --check` and worktree status are clean. - Read `CLAUDE.md` and the round-4 review log. - Confirmed the required clean starting HEAD and branch. - Confirmed the local GitNexus index is absent; direct AST/source tracing will @@ -160,6 +170,5 @@ No push or external mutation is authorized. ## Next -- Run the full `populace-build` suite and repository-wide Ruff. - Write `/private/tmp/583_fix3_handoff.md`, then restore `PROGRESS.md` to `origin/main` and commit that restoration so root journals stay out of the PR. From 3a404e884df242445238e162db547877747e03a4 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 01:22:32 -0400 Subject: [PATCH 54/94] Keep PROGRESS.md at origin/main --- PROGRESS.md | 204 ++++++++++------------------------------------------ 1 file changed, 39 insertions(+), 165 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index f73dbfbb..f3158fdd 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,174 +1,48 @@ -# PR #583 Fix-3 Guard Completion Progress +# Progress ## State -Fix-3 completion is in progress on `multispine-pool-build-578`, from clean -starting commit `b219b21`. The populace owner resolved the typed-DataFrame -decision: parameterized `df[column]` is a permitted boundary, while every -statically resolvable guarded column name is contraband in every expression -position in a non-owner operator module. The three-layer contract, composition -argument, sole runtime-data residual, and annotation/docstring exemptions are -now documented and executable. All requested binding tests pass, and both the -all-runtime scan and pinned 54-module graph are clean after three narrow -adjudications. Final guard/package/Ruff certification is green. The fix-3 -handoff and required root-journal restoration remain. - -No push or external mutation is authorized. +Populace #516 whole-row donor outlier screen is complete on +`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515 +interim carve merged as #525). The `puf_tax_detail` donor now drops tax units +whose grouped raw mortgage interest reaches $10M before the #515 carve +(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T +of phantom mortgage-interest mass), with the checkpoint schema bumped to v3 +so post-carve pre-screen checkpoints rebuild. ## Done -- Read `CLAUDE.md` and `/private/tmp/583_fix2_handoff.md`. -- Confirmed the required branch, clean worktree, and starting HEAD `b219b21`. -- Recorded the owner's explicit typed-parameter and contraband-literal decision. -- Added the all-expression contraband rule on top of the existing flow-sensitive - string resolver, including static f-string alternatives from enumerated - comprehension choices. -- Exempted only true module/class/function docstrings and annotation syntax; - executable defaults, including lambda defaults, remain checked. -- Added binding tests for the typed parameter boundary, guarded and benign - helper calls, list/tuple/set/dict positions, comparisons, returns, - assignments, defaults, every requested composition form, static f-string - enumeration, owners, annotations, and docstrings. -- Classified `operator_boundary.py` as a reviewed provenance owner because it - enumerates provenance columns only to reject preassembled source frames. -- Moved the shared `person_support_channel` spelling into the existing - `support_provenance.py` owner and imported it into `adult_care.py` and - `ssi_take_up.py`, leaving both treatment modules fully guarded. -- Ran the guard file: 94 passed, with the all-runtime and exact 54-module graph - scans clean. Focused Ruff and `git diff --check` pass. -- Closed post-implementation adversarial findings: function defaults and - decorators now resolve at definition time while bodies retain late-binding - analysis; class namespaces no longer corrupt enclosing constants; PEP 695 - class type parameters remain annotation-exempt; and attribute assignment, - annotated-assignment, loop, and comprehension targets are visited. -- Documented the full non-strict receiver composition boundary for typed, - constructed-DataFrame, and `Frame.table(...)` result subscripts, with no - interprocedural proof and the same sole runtime-data residual. -- Added binding regressions for definition-time constant state, class namespace - isolation, PEP 695 annotations, every attribute-store form, and the broader - truthful subscript boundary. -- Preserved real nested lambda/comprehension closures while excluding class - namespaces, and distinguished abstract static iteration choices so composed - fragments resolve across those closures without treating ordinary tuple - formatting as alternative values. -- The expanded guard file passes 99 tests; the all-runtime and exact 54-module - scans remain clean. -- Completed independent adversarial review with a final clean verdict after - binding every reported definition-time, class-scope, annotation, store, - closure, and static-format precision repro. -- Ran the final guard file through `uv`: 99 passed. -- Ran the final full `populace-build` suite: 3,321 passed, 85 skipped, 5 known - warnings in 107.66 seconds. -- Ran repository-wide `ruff check .`: passed. -- Ran focused `ruff format --check` on all four changed code/test files: - passed. `git diff --check` and worktree status are clean. -- Read `CLAUDE.md` and the round-4 review log. -- Confirmed the required clean starting HEAD and branch. -- Confirmed the local GitNexus index is absent; direct AST/source tracing will - be used unless a safe local index becomes available. -- Completed parallel read-only audits of reviewer repros, resolver/loop - behavior, and strict call-site/closure behavior. -- Extended the static resolver for walrus expressions, string multiplication, - percent formatting, and all-static chained `str.replace`. -- Made inferred column-container subscripts resolve every static string member - or record explicit opacity; subscript assignment targets are visited too. -- Added exact round-4 binding tests: walrus binds and reports both reads; - multiplication, percent formatting, and replace report the guarded name; - nested calls and dict indirection report fail-closed. -- Replayed the guard file (15 passed), focused Ruff, and the exact 54-module - graph (no missing modules or offenders). -- Bound `for`/`async for` targets and list, set, dict, and generator - comprehension targets before their bodies are visited. -- Represented statically resolvable iterables as tuples of every possible - string, checking every member at a subscript; dynamic iterables explicitly - shadow stale outer bindings with opacity. -- Added benign, mixed guarded, module-bound, dynamic, and comprehension-wrapped - selector regressions. The guard file now passes 17 tests with the graph clean. -- Replaced bare `{}` substitution with `string.Formatter` field parsing for - automatic, indexed, named, converted (`!s`/`!r`), and specified fields. -- Kept unresolved fields structurally opaque while preserving literal `*` in a - fully static pandas expression as benign syntax. -- Bound exact-name checks for every unsafe format variant and zero-finding - controls for named benign formatting, conversions/specs, escaped braces, and - multiplication syntax. The guard file now passes 19 tests with the graph - clean. -- Tracked scoped aliases of `query`, `eval`, `filter`, and `get`, including - alias chains, explicit rebinding shadows, and static `getattr` aliases. -- Routed direct and aliased calls through shared strict handlers, including - hidden/expanded argument and opaque-key failures. -- Made dynamic `getattr` fail closed for inferred DataFrame/Frame containers - while distinguishing generic object and Series index access. -- Added direct-vs-alias exact-name tests for all four methods, benign and opaque - batteries, rebinding/parameter-shadow controls, and static/dynamic `getattr` - probes. The guard file now passes 22 tests with the 54-module graph clean. -- Added a lexical scope pre-pass that counts binding sites without descending - nested function/class bodies and pre-shadows all Python locals. -- Deferred nested function-body analysis until the enclosing scope's bindings - are complete; free names with exactly one defining-scope assignment resolve, - while multi-assignment free names become explicit opacity. -- Preserved the same late-binding rule for lambdas and strict method aliases, - so a rebound alias cannot degrade to an ignored ordinary Name call. -- Added the exact reviewer late-bound closure, module-level and lambda variants, - stable guarded/benign controls, later-local shadowing, and stable/rebound - alias tests. The guard file now passes 24 tests with the graph clean. -- Replaced the guard module and scan-test docstrings with the completed - resolve-and-check-or-fail-closed contract and an explicit surface inventory. -- Added one parametrized invariant covering bound names, parameter f-strings, - expanded kwargs, shadows, conditionals, mutation, all requested format - variants, all six round-4 subscript evasions, all four method aliases, - dynamic `getattr`, and late closure binding. -- Added a parametrized benign battery spanning every corresponding static - surface and typed non-column controls. -- Pinned the multispine tool graph to exactly 54 runtime modules. -- Replayed the expanded guard file (70 passed), focused Ruff, and - `git diff --check`; the exact 54-module graph remains clean. -- Preserved strict method identity through walrus expressions and structural - tuple/list bindings; a previously strict alias rebound to an unresolved - callable now becomes an explicit opaque alias rather than disappearing. -- Tracked aliases of builtin `getattr` and expanded literal starred arguments; - unresolved starred calls fail closed. -- Added guarded-name, benign, rebound, unpacked, aliased-`getattr`, and starred - `getattr` binding tests. The guard file now passes 71 tests with the graph - clean. -- Expanded statically known `*args`/`**kwargs` before assigning format field - positions, while any unresolved expansion that can supply a referenced field - becomes opaque. -- Recursively resolved nested format specifications and static collection - indexing, and resolved static mapping operands for percent formatting. -- Added exact-name and benign controls for nested specs, indexed mappings, - static positional/keyword expansion, and percent mappings, plus the - adversarial star-index-shift failure. The guard remains 71 tests green with - the graph clean. -- Counted assignment-expression targets inside comprehensions in their actual - containing Python scope and routed their runtime bindings past the synthetic - comprehension scope. -- Counted nested `nonlocal`/`global` stores against the lexical scope they can - rebind, while excluding those declarations from the nested function's local - binding count. -- Added direct subscript and closure tests for comprehension walruses plus - sibling `nonlocal` and module `global` writers. The guard remains 71 tests - green with the graph clean. -- Generalized static string iteration from list/tuple syntax to sets, dict - keys, literal strings, and statically concatenated collections while - preserving literal containers for format-field indexing. -- Added benign and guarded loop/comprehension controls for every new iterable - form. The guard file now passes 72 tests with the graph clean. -- Added branch-state joins for `if` statements/expressions and zero-or-more - iteration joins for dynamic loops; divergent constants become opaque, - possible DataFrame provenance is retained, and conditional method aliases - cannot disappear. -- Preserved exact pre-loop bindings across provably empty static iterables. -- Added guarded, benign, conditional-alias, optional-loop, and empty-loop - binding tests. The guard file now passes 73 tests with the graph clean. -- Ran the full `populace-build` suite: 3,295 passed, 85 skipped, 5 warnings. -- Ran repository-wide `ruff check .`: passed. -- Ran repository-wide `ruff format --check .`: found the existing 44-file - formatting baseline; formatted only this guard file and replayed its 73 tests. -- Wrote `/private/tmp/583_fix2_handoff.md` with per-subsystem rules and binding - tests, the no-rewrite rationale, exact validation receipts, commit inventory, - and the remaining typed-container blocker. +- Confirmed a clean starting worktree at `aef1c56`. +- Read the repository guidance and established the #515 donor carve as the + screen's required downstream boundary. +- Started source-level audits of every donor-frame consumer, checkpoint + validation, row-count pins, and existing donor-fact summaries. +- Attempted the requested GitNexus impact workflow; the managed filesystem + denied its global registry write. Its local index also exposed a broad + `build/` ignore mismatch, so the completed impact audit uses direct source + call sites and tests. +- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the + structural rationale and pinned-artifact receipts. +- Added a whole-row screen on grouped raw person `home_mortgage_interest` + after tax-unit assembly, before the #515 carve, with retained-index reset. +- Confirmed no downstream consumer pairs donor rows to the original HDF arrays + or carries a stale donor-length vector; values and weights always originate + from the same screened frame. +- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale + checkpoint regression track the live constant while retaining literal-v1 + corruptions. +- Added regression coverage for the exact grouped boundary, whole-row removal, + retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant. +- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets + 139; populace-data 138 with 1 skip. The directly affected tail-bound suite + adds 12 passes. Ruff format/check and `git diff --check` are clean. +- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line + audit, expected 208,611-row real-artifact effect, verification results, count + sweep, and deliberately untouched surfaces. ## Next -- Write `/private/tmp/583_fix3_handoff.md`, then restore `PROGRESS.md` to - `origin/main` and commit that restoration so root journals stay out of the PR. +- PR #527 review cycle, then merge. After both #525 and #527: rebuild the + base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a + run that holds per `us_critical_targets.py`. +- Root record-level ETL carve stays open on populace#515. From dd6385883cd976892870b9a9d9ee67b335c119f2 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 01:43:56 -0400 Subject: [PATCH 55/94] Start PR 583 final guard journal --- PROGRESS.md | 56 +++++++++++++++++------------------------------------ 1 file changed, 18 insertions(+), 38 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index f3158fdd..8e6710d7 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,47 +2,27 @@ ## State -Populace #516 whole-row donor outlier screen is complete on -`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515 -interim carve merged as #525). The `puf_tax_detail` donor now drops tax units -whose grouped raw mortgage interest reaches $10M before the #515 carve -(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T -of phantom mortgage-interest mass), with the checkpoint schema bumped to v3 -so post-carve pre-screen checkpoints rebuild. +PR #583 final guard remediation is in progress on +`multispine-pool-build-578`, starting from clean HEAD `3a404e8`. Round 5 held +certification because the guard's completeness claim exceeded what a static +Python scanner can honestly guarantee. The owner has narrowed the contract to +enumerated certified surfaces with named adversarial and runtime-data +boundaries. ## Done -- Confirmed a clean starting worktree at `aef1c56`. -- Read the repository guidance and established the #515 donor carve as the - screen's required downstream boundary. -- Started source-level audits of every donor-frame consumer, checkpoint - validation, row-count pins, and existing donor-fact summaries. -- Attempted the requested GitNexus impact workflow; the managed filesystem - denied its global registry write. Its local index also exposed a broad - `build/` ignore mismatch, so the completed impact audit uses direct source - call sites and tests. -- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the - structural rationale and pinned-artifact receipts. -- Added a whole-row screen on grouped raw person `home_mortgage_interest` - after tax-unit assembly, before the #515 carve, with retained-index reset. -- Confirmed no downstream consumer pairs donor rows to the original HDF arrays - or carries a stale donor-length vector; values and weights always originate - from the same screened frame. -- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale - checkpoint regression track the live constant while retaining literal-v1 - corruptions. -- Added regression coverage for the exact grouped boundary, whole-row removal, - retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant. -- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets - 139; populace-data 138 with 1 skip. The directly affected tail-bound suite - adds 12 passes. Ruff format/check and `git diff --check` are clean. -- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line - audit, expected 208,611-row real-artifact effect, verification results, count - sweep, and deliberately untouched surfaces. +- Read `CLAUDE.md` and the PR-review remediation workflow. +- Confirmed the requested branch, commit, clean worktree, and round-5 HOLD + receipt at `/Users/maxghenis/PolicyEngine/_buildo-runtime/reviews/sol_583_r5.log`. +- Clustered the requested work into contract narrowing, four natural-class + catches, explicit pinned out-of-scope misses, runtime constant-plumbing + verification, and handoff accuracy. ## Next -- PR #527 review cycle, then merge. After both #525 and #527: rebuild the - base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a - run that holds per `us_critical_targets.py`. -- Root record-level ETL carve stays open on populace#515. +- Inspect the guard implementation and historical adversarial regressions. +- Implement and commit the narrowed contract plus natural-class catches. +- Pin the documented evasions as intentional misses. +- Run the guard file, full `populace-build` suite, Ruff, and value-preservation + checks; write `/private/tmp/583_fix4_handoff.md`. +- Restore `PROGRESS.md` byte-for-byte to `origin/main` before final handoff. From 5b685defa6d8bad189554dcf2e103cb465e4ff71 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 01:58:00 -0400 Subject: [PATCH 56/94] Reframe and close PR 583 guard scope --- PROGRESS.md | 37 +- .../tests/test_us_spine_blindness.py | 481 +++++++++++++++--- 2 files changed, 437 insertions(+), 81 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 8e6710d7..fbe1d21f 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,12 +2,11 @@ ## State -PR #583 final guard remediation is in progress on -`multispine-pool-build-578`, starting from clean HEAD `3a404e8`. Round 5 held -certification because the guard's completeness claim exceeded what a static -Python scanner can honestly guarantee. The owner has narrowed the contract to -enumerated certified surfaces with named adversarial and runtime-data -boundaries. +PR #583 final guard remediation is implemented on +`multispine-pool-build-578`. The contract now certifies only enumerated static +surfaces and names adversarial construction and runtime-materialized names as +explicit boundaries. The full guard lane is green, including every runtime +module and the pinned 54-module build graph. ## Done @@ -17,12 +16,28 @@ boundaries. - Clustered the requested work into contract narrowing, four natural-class catches, explicit pinned out-of-scope misses, runtime constant-plumbing verification, and handoff accuracy. +- Replaced the unsupportable completeness claim with the owner-directed + tripwire contract and its two named out-of-scope classes. +- Added static folding for `lower`, `upper`, `casefold`, `title`, and + `capitalize`. +- Propagated all static loop/comprehension choices through `str.format` and + f-string interpolation; the exact `for-entity-format` repro now reports both + guarded columns by name. +- Routed direct `df.__getitem__(column)` through the same selector analysis as + `df[column]`. +- Added named-expression factory alias discovery; strict-method walrus + aliasing is pinned for both guarded and opaque expressions. +- Added the exact round-5 reverse-slice, `format_map`, `__doc__`, + `__annotations__`, and container-indexed-method repros to + `test_documented_out_of_scope_evasions_are_not_caught`, asserting their + intentional current misses. +- Passed the complete guard file: 113 tests. ## Next -- Inspect the guard implementation and historical adversarial regressions. -- Implement and commit the narrowed contract plus natural-class catches. -- Pin the documented evasions as intentional misses. -- Run the guard file, full `populace-build` suite, Ruff, and value-preservation - checks; write `/private/tmp/583_fix4_handoff.md`. +- Commit the completed guard contract, implementation, regressions, and this + receipt. +- Reverify adult-care and SSI constant-plumbing value preservation. +- Run the full `populace-build` suite and Ruff; update the PR body and write + `/private/tmp/583_fix4_handoff.md`. - Restore `PROGRESS.md` byte-for-byte to `origin/main` before final handoff. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 94eb324a..99668b42 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -1,29 +1,29 @@ -"""Completed fail-closed guard for source-spine-blind US operators. - -The contract has three layers in every non-owner operator module: - -1. Every column-access surface kind is visited, and every locally strict - occurrence resolves to static choices that are checked by name or fails - closed: subscripts and ``.loc`` reads/writes, attributes, ``getattr``, - canonical column factories, and direct or aliased pandas ``get``, - ``filter``, ``query``, and ``eval`` calls. -2. Opacity is a violation at every strict surface; unresolved selectors, - expressions, aliases, and expanded arguments never create a silent state. -3. Guarded column names are contraband anywhere statically visible in an - executable expression in a non-owner module, including calls, containers, - comparisons, returns, defaults, and assignments. - -A dynamic ``helper(df, column)`` can reach a guarded column only through its -name, and that name must be written somewhere. Wherever it is statically -written, layer 3 catches it using the same constant and string-composition -resolution as the access checks. Therefore a typed parameter-to-parameter -``df[column]`` subscript is deliberately permitted: its producing site bears -the name check. Constructed-DataFrame and ``Frame.table(...)`` result -subscripts outside local strict-receiver inference use the same composition -boundary. No interprocedural proof is attempted. The single accepted residual -is a guarded name materialized purely from runtime file, configuration, or -environment content, where no static spelling exists to inspect. Annotations -and docstrings are exempt because they are not dataflow. +"""Tripwire against accidental spine-awareness, not an adversarial sandbox. + +This guard enforces, and its tests certify, exactly these surfaces: + +- direct attribute, subscript, ``.loc``, and ``__getitem__`` reads; +- canonical guarded-column factory calls, including aliases bound by simple + assignment or named expression; +- ``query``, ``eval``, ``filter``, and ``get`` expression surfaces, failing + closed on opacity including hidden arguments and method aliases bound by + simple assignment or named expression; +- one-level static indirection through constants, concatenation, f-strings, + ``str.format`` with full field syntax, ``%`` formatting, ``str * int``, + static ``.replace`` chains, and the static-receiver case methods ``lower``, + ``upper``, ``casefold``, ``title``, and ``capitalize``; +- loop and comprehension propagation over static iterables; and +- contraband guarded-name literals anywhere statically visible in non-owner + modules. + +Two classes are out of scope by design, and naming them is the honest +boundary. First, deliberately obfuscated construction -- reverse slicing, +``format_map`` over dynamic maps, ``__doc__`` or ``__annotations__`` mining, +container-indexed method aliases, and kin -- is controlled by code review and +the adversarial merge-review process. A scanner that claimed to catch code +written to deceive would be lying. Second, column names materialized purely +from runtime data are controlled by the assembly receipt and runtime +validation. """ from __future__ import annotations @@ -31,6 +31,7 @@ import ast import fnmatch import re +from itertools import product from pathlib import Path from string import Formatter @@ -219,10 +220,13 @@ def _call_name(node: ast.Call) -> str | None: - if isinstance(node.func, ast.Name): - return node.func.id - if isinstance(node.func, ast.Attribute): - return node.func.attr + function = node.func + if isinstance(function, ast.NamedExpr): + return function.target.id + if isinstance(function, ast.Name): + return function.id + if isinstance(function, ast.Attribute): + return function.attr return None @@ -300,12 +304,18 @@ def _factory_aliases(tree: ast.AST) -> set[str]: while changed: changed = False for node in ast.walk(tree): - if not isinstance(node, (ast.Assign, ast.AnnAssign)): + if isinstance(node, ast.NamedExpr): + value = node.value + targets = [node.target] + elif isinstance(node, (ast.Assign, ast.AnnAssign)): + value = node.value + targets = ( + node.targets if isinstance(node, ast.Assign) else [node.target] + ) + else: continue - value = node.value if not isinstance(value, ast.Name) or value.id not in aliases: continue - targets = node.targets if isinstance(node, ast.Assign) else [node.target] for target in targets: if isinstance(target, ast.Name) and target.id not in aliases: aliases.add(target.id) @@ -412,6 +422,24 @@ def _static_string_shape( except (TypeError, ValueError): return None return result if isinstance(result, str) else None + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and node.func.attr in {"lower", "upper", "casefold", "title", "capitalize"} + and not node.args + and not node.keywords + ): + value = _static_string_shape(node.func.value, constants) + if value is None: + return None + case_method = { + "lower": str.lower, + "upper": str.upper, + "casefold": str.casefold, + "title": str.title, + "capitalize": str.capitalize, + }[node.func.attr] + return case_method(value) if ( isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) @@ -713,15 +741,52 @@ def _static_string_list( if values is None: return None for name in _assigned_names(generator.target): - local[name] = values + local[name] = _StaticStringChoices(values) return _static_string_values(node.elt, nested_constants) return None +def _active_static_string_choices( + node: ast.AST, + constants: list[dict[str, object]], +) -> tuple[tuple[str, _StaticStringChoices], ...]: + """Find loaded loop/comprehension choices used by one expression.""" + + choices: list[tuple[str, _StaticStringChoices]] = [] + names = { + child.id + for child in ast.walk(node) + if isinstance(child, ast.Name) and isinstance(child.ctx, ast.Load) + } + for name in sorted(names): + for scope in reversed(constants): + if name not in scope: + continue + value = scope[name] + if isinstance(value, _StaticStringChoices): + choices.append((name, value)) + break + return tuple(choices) + + def _static_string_values( node: ast.AST, constants: list[dict[str, object]], ) -> tuple[str, ...] | None: + active_choices = _active_static_string_choices(node, constants) + if active_choices: + resolved: list[str] = [] + names = tuple(name for name, _ in active_choices) + alternatives = tuple(tuple(values) for _, values in active_choices) + for selected in product(*alternatives): + values = _static_string_values( + node, + [*constants, dict(zip(names, selected, strict=True))], + ) + if values is None: + return None + resolved.extend(values) + return tuple(dict.fromkeys(resolved)) if isinstance(node, ast.JoinedStr): alternatives = _static_joined_string_values(node, constants) if alternatives is not None: @@ -1976,6 +2041,69 @@ def _visit_strict_method_call( else: self._visit_filter_call(node) + def _visit_column_selector( + self, + node: ast.AST, + *, + receiver: ast.AST, + selector: ast.AST, + ) -> None: + """Apply identical selector checks to [] and direct __getitem__.""" + + column_container = self._column_container(receiver) + resolved = _static_string_values(selector, self.constants) + column = _subscript_source_expression( + selector, + bindings=self.bindings, + factory_aliases=self.factory_aliases, + ) + if resolved is None: + if column is not None: + self._record(node, f"subscript using {column}") + elif column_container: + self._record( + node, + "subscript with an unresolvable dynamic selector (fail-closed)", + ) + elif any(_OPAQUE_STRING_PART in item for item in resolved): + if ( + column_container + or column is not None + or any(_is_source_column_shape(item) for item in resolved) + ): + self._record( + node, + "subscript with an unresolvable dynamic selector (fail-closed)", + ) + else: + for item in resolved: + if _is_source_column_shape(item): + self._record( + node, + f"subscript using source column {item!r}", + ) + + def _visit_getitem_call(self, node: ast.Call) -> None: + """Treat a direct receiver.__getitem__(key) exactly like receiver[key].""" + + receiver = node.func.value + if ( + node.keywords + or len(node.args) != 1 + or isinstance(node.args[0], ast.Starred) + ): + if self._column_container(receiver): + self._record( + node, + "__getitem__() with hidden or expanded arguments (fail-closed)", + ) + return + self._visit_column_selector( + node, + receiver=receiver, + selector=node.args[0], + ) + def visit_Call(self, node: ast.Call) -> None: if ( isinstance(node.func, ast.Attribute) @@ -1983,6 +2111,8 @@ def visit_Call(self, node: ast.Call) -> None: and node.func.attr in self._MUTATORS ): self._poison(node.func.value.id) + if isinstance(node.func, ast.Attribute) and node.func.attr == "__getitem__": + self._visit_getitem_call(node) name = _call_name(node) if name in self.factory_aliases: self._record(node, f"call to {name}") @@ -2010,7 +2140,7 @@ def visit_Attribute(self, node: ast.Attribute) -> None: def visit_Subscript(self, node: ast.Subscript) -> None: selector = node.slice - column_container = self._column_container(node.value) + receiver = node.value if ( isinstance(node.value, ast.Attribute) and node.value.attr == "loc" @@ -2018,39 +2148,12 @@ def visit_Subscript(self, node: ast.Subscript) -> None: and node.slice.elts ): selector = node.slice.elts[-1] - column_container = self._column_container(node.value.value) - - resolved = _static_string_values(selector, self.constants) - column = _subscript_source_expression( - selector, - bindings=self.bindings, - factory_aliases=self.factory_aliases, + receiver = node.value.value + self._visit_column_selector( + node, + receiver=receiver, + selector=selector, ) - if resolved is None: - if column is not None: - self._record(node, f"subscript using {column}") - elif column_container: - self._record( - node, - "subscript with an unresolvable dynamic selector (fail-closed)", - ) - elif any(_OPAQUE_STRING_PART in item for item in resolved): - if ( - column_container - or column is not None - or any(_is_source_column_shape(item) for item in resolved) - ): - self._record( - node, - "subscript with an unresolvable dynamic selector (fail-closed)", - ) - else: - for item in resolved: - if _is_source_column_shape(item): - self._record( - node, - f"subscript using source column {item!r}", - ) self.generic_visit(node) @@ -2407,6 +2510,26 @@ def op(df): assert _source_spine_accesses(raw_spine_source_id) +def test_named_expression_factory_alias_is_caught() -> None: + """A walrus-bound canonical factory remains a named factory call.""" + + immediate = """ +def f(): + return (factory := support_channel_column)("person") +""" + later = """ +def f(): + (factory := support_channel_column) + return factory("person") +""" + + for source in (immediate, later): + accesses = _source_spine_accesses(source) + assert accesses, source + assert any("call to factory" in access for access in accesses) + assert all("fail-closed" not in access for access in accesses) + + def test_source_spine_ast_guard_detects_pandas_string_reads() -> None: """Pandas string-based column APIs cannot bypass the structural guard.""" @@ -2688,6 +2811,32 @@ def test_contraband_names_reuse_static_expression_resolution( assert all("fail-closed" not in access for access in accesses) +@pytest.mark.parametrize( + ("case_method", "expression"), + ( + ("lower", '"PERSON_SUPPORT_CHANNEL".lower()'), + ("upper", '"pErSoN_sUpPoRt_cHaNnEl".upper().lower()'), + ("casefold", '"PERSON_SUPPORT_CHANNEL".casefold()'), + ( + "title", + '"PERSON SUPPORT CHANNEL".title().replace(" ", "_").lower()', + ), + ("capitalize", '"PERSON_SUPPORT_CHANNEL".capitalize().lower()'), + ), +) +def test_static_case_methods_fold_guarded_names( + case_method: str, + expression: str, +) -> None: + """Every documented zero-argument case method resolves static receivers.""" + + source = f"def f():\n return sink({expression})\n" + accesses = _source_spine_accesses(source) + assert accesses, case_method + assert any("person_support_channel" in access for access in accesses) + assert all("fail-closed" not in access for access in accesses) + + def test_contraband_names_resolve_bound_and_enumerated_fragments() -> None: """Bound concatenation and static f-string choices expose exact names.""" @@ -2763,6 +2912,55 @@ def f(): assert _source_spine_accesses(later_expression) +def test_documented_out_of_scope_evasions_are_not_caught() -> None: + """Pin deliberate obfuscation outside this accidental-awareness tripwire. + + Code review and adversarial merge review, rather than this scanner, control + code written to deceive. Preserving these known misses keeps that boundary + visible and prevents claims that the scanner is an adversarial sandbox. + """ + + sources = { + "reverse slicing": """import pandas as pd +def select(df: pd.DataFrame, column: str): + return df[column] +def op(df): + column = "nosrep"[::-1] + "_" + "troppus"[::-1] + "_" + "lennahc"[::-1] + return select(df, column) +""", + "format_map": """import pandas as pd +def select(df: pd.DataFrame, column: str): + return df[column] +def op(df): + column = "{a}_{b}".format_map({"a": "person", "b": "support_channel"}) + return select(df, column) +""", + "__doc__ mining": '''import pandas as pd +def marker(): + """person_support_channel""" +def select(df: pd.DataFrame, column: str): + return df[column] +def op(df): + return select(df, marker.__doc__) +''', + "__annotations__ mining": """import pandas as pd +class Marker: + person_support_channel: str +def select(df: pd.DataFrame, column: str): + return df[column] +def op(df): + return select(df, next(iter(Marker.__annotations__))) +""", + "container-indexed strict alias": """def f(df, expr): + query = (df.query,)[0] + return query(expr) +""", + } + + for evasion, source in sources.items(): + assert _source_spine_accesses(source) == (), evasion + + def test_attribute_store_surfaces_are_always_visited() -> None: """Assignments and loop targets cannot hide guarded attribute writes.""" @@ -3014,6 +3212,61 @@ def f(df): assert any("fail-closed" in access for access in accesses) +def test_dunder_getitem_matches_subscript_selector_checks() -> None: + """Direct __getitem__ calls have the same static and opacity boundary.""" + + equivalent_sources = ( + ( + """ +def f(df): + return df["person_support_channel"] +""", + """ +def f(df): + return df.__getitem__("person_support_channel") +""", + "person_support_channel", + ), + ( + """ +def f(df, column): + return df[column] +""", + """ +def f(df, column): + return df.__getitem__(column) +""", + "fail-closed", + ), + ) + benign_sources = ( + """ +def f(df): + return df["age"] +""", + """ +def f(df): + return df.__getitem__("age") +""", + """ +def f(df: pd.DataFrame, column: str): + return df[column] +""", + """ +def f(df: pd.DataFrame, column: str): + return df.__getitem__(column) +""", + ) + + for subscript, dunder, expected in equivalent_sources: + for source in (subscript, dunder): + accesses = _source_spine_accesses(source) + assert accesses, source + assert any(expected in access for access in accesses) + for source in benign_sources: + assert _source_spine_accesses(source) == (), source + + def test_static_selector_extensions_are_shared_by_strict_pandas_calls() -> None: """Mult, percent formatting, and replace resolve at every strict surface.""" @@ -3192,6 +3445,46 @@ def f(df): assert any("fail-closed" in item for item in accesses) +def test_for_entity_format_repro_is_caught_by_name() -> None: + """Loop/comprehension format and f-string interpolation expand all choices.""" + + sources = { + "for-entity-format": """ +def f(): + for entity in ("person", "household"): + sink("{}_support_channel".format(entity)) +""", + "for-entity-fstring": """ +def f(): + for entity in ("person", "household"): + sink(f"{entity}_support_channel") +""", + "comprehension-entity-format": """ +def f(): + return [ + "{}_support_channel".format(entity) + for entity in ("person", "household") + ] +""", + "comprehension-entity-fstring": """ +def f(): + return [ + f"{entity}_support_channel" + for entity in ("person", "household") + ] +""", + } + + for construction, source in sources.items(): + accesses = _source_spine_accesses(source) + for column in ( + "person_support_channel", + "household_support_channel", + ): + assert any(column in access for access in accesses), construction + assert all("fail-closed" not in access for access in accesses) + + def test_format_fields_resolve_all_static_forms_by_name() -> None: """Automatic, indexed, named, converted, and specified fields are exact.""" @@ -3448,7 +3741,7 @@ def test_strict_method_aliases_cover_expression_and_structural_bindings() -> Non guarded_sources = ( """ def f(df): - return (query := df.query)("person_support_channel == 1") + return (q := df.query)("person_support_channel == 1") """, """ def f(df): @@ -3459,7 +3752,7 @@ def f(df): benign_sources = ( """ def f(df): - return (query := df.query)("age >= 18") + return (q := df.query)("age >= 18") """, """ def f(df): @@ -3475,6 +3768,14 @@ def f(df): for source in benign_sources: assert _source_spine_accesses(source) == (), source + opaque = """ +def f(df, expr): + return (q := df.query)(expr) +""" + opaque_accesses = _source_spine_accesses(opaque) + assert opaque_accesses + assert any("fail-closed" in access for access in opaque_accesses) + def test_getattr_column_access_resolves_or_fails_closed() -> None: """Static attributes are checked by name; dynamic table attrs are opaque.""" @@ -3871,12 +4172,52 @@ def inner(): return df.query(expr) expr = "person_support_channel == 1" return inner() +""", + ), + ( + "round5-static-lower", + """ +import pandas as pd +def select(df: pd.DataFrame, column: str): + return df[column] +def op(df): + column = "PERSON_SUPPORT_CHANNEL".lower() + return select(df, column) +""", + ), + ( + "round5-for-entity-format", + """ +def f(): + for entity in ("person", "household"): + sink("{}_support_channel".format(entity)) +""", + ), + ( + "round5-dunder-getitem", + """ +def f(df, column): + return df.__getitem__(column) +""", + ), + ( + "round5-factory-walrus", + """ +def f(): + return (factory := support_channel_column)("person") +""", + ), + ( + "round5-query-walrus", + """ +def f(df, expr): + return (q := df.query)(expr) """, ), ), ) def test_every_review_evasion_is_caught(evasion: str, source: str) -> None: - """Rounds 2-4 are one permanent resolve-or-fail-closed invariant.""" + """Enumerated in-scope rounds 2-5 cases resolve or fail closed.""" assert _source_spine_accesses(source), evasion From ae7fbc1b379628b0adf57186d3176d792dd079fc Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 01:58:31 -0400 Subject: [PATCH 57/94] Record PR 583 value preservation checks --- PROGRESS.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index fbe1d21f..87a66cfd 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -32,12 +32,14 @@ module and the pinned 54-module build graph. `test_documented_out_of_scope_evasions_are_not_caught`, asserting their intentional current misses. - Passed the complete guard file: 113 tests. +- Reverified the value-preserving runtime constant plumbing with the full + adult-care and SSI files: 85 passed, 2 skipped. `support_provenance.py` owns + the unchanged `"person_support_channel"` value; guarded consumers + `adult_care.py` and `ssi_take_up.py` import it without behavior changes. +- Passed repository-wide Ruff and `git diff --check`. ## Next -- Commit the completed guard contract, implementation, regressions, and this - receipt. -- Reverify adult-care and SSI constant-plumbing value preservation. -- Run the full `populace-build` suite and Ruff; update the PR body and write - `/private/tmp/583_fix4_handoff.md`. +- Run the full `populace-build` suite; update the PR body and write + `/private/tmp/583_fix4_handoff.md` with exact receipts. - Restore `PROGRESS.md` byte-for-byte to `origin/main` before final handoff. From 92f97f4eb868f5283c47fedf47c1a076ebf44683 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 02:01:42 -0400 Subject: [PATCH 58/94] Record PR 583 final suite receipt --- PROGRESS.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 87a66cfd..65db1586 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -37,9 +37,16 @@ module and the pinned 54-module build graph. the unchanged `"person_support_channel"` value; guarded consumers `adult_care.py` and `ssi_take_up.py` import it without behavior changes. - Passed repository-wide Ruff and `git diff --check`. +- Passed the full `populace-build` suite: 3,335 passed, 85 skipped, with five + pre-existing warnings, in 107.47 seconds. +- Prepared an accurate replacement PR body covering the enumerated contract, + the sole `operator_boundary.py` owner addition, and all three runtime + constant-plumbing files. The live connector update was canceled, so the + replacement will be preserved verbatim in the final handoff rather than + applied through another channel. ## Next -- Run the full `populace-build` suite; update the PR body and write - `/private/tmp/583_fix4_handoff.md` with exact receipts. +- Write `/private/tmp/583_fix4_handoff.md` with exact receipts and the + unapplied PR-body replacement. - Restore `PROGRESS.md` byte-for-byte to `origin/main` before final handoff. From cb058c0d0b69855b7a87d1a32251ccff969243ac Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 02:02:55 -0400 Subject: [PATCH 59/94] Keep PROGRESS.md at origin/main after fix 4 --- PROGRESS.md | 80 +++++++++++++++++++++++++---------------------------- 1 file changed, 38 insertions(+), 42 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 65db1586..f3158fdd 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,51 +2,47 @@ ## State -PR #583 final guard remediation is implemented on -`multispine-pool-build-578`. The contract now certifies only enumerated static -surfaces and names adversarial construction and runtime-materialized names as -explicit boundaries. The full guard lane is green, including every runtime -module and the pinned 54-module build graph. +Populace #516 whole-row donor outlier screen is complete on +`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515 +interim carve merged as #525). The `puf_tax_detail` donor now drops tax units +whose grouped raw mortgage interest reaches $10M before the #515 carve +(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T +of phantom mortgage-interest mass), with the checkpoint schema bumped to v3 +so post-carve pre-screen checkpoints rebuild. ## Done -- Read `CLAUDE.md` and the PR-review remediation workflow. -- Confirmed the requested branch, commit, clean worktree, and round-5 HOLD - receipt at `/Users/maxghenis/PolicyEngine/_buildo-runtime/reviews/sol_583_r5.log`. -- Clustered the requested work into contract narrowing, four natural-class - catches, explicit pinned out-of-scope misses, runtime constant-plumbing - verification, and handoff accuracy. -- Replaced the unsupportable completeness claim with the owner-directed - tripwire contract and its two named out-of-scope classes. -- Added static folding for `lower`, `upper`, `casefold`, `title`, and - `capitalize`. -- Propagated all static loop/comprehension choices through `str.format` and - f-string interpolation; the exact `for-entity-format` repro now reports both - guarded columns by name. -- Routed direct `df.__getitem__(column)` through the same selector analysis as - `df[column]`. -- Added named-expression factory alias discovery; strict-method walrus - aliasing is pinned for both guarded and opaque expressions. -- Added the exact round-5 reverse-slice, `format_map`, `__doc__`, - `__annotations__`, and container-indexed-method repros to - `test_documented_out_of_scope_evasions_are_not_caught`, asserting their - intentional current misses. -- Passed the complete guard file: 113 tests. -- Reverified the value-preserving runtime constant plumbing with the full - adult-care and SSI files: 85 passed, 2 skipped. `support_provenance.py` owns - the unchanged `"person_support_channel"` value; guarded consumers - `adult_care.py` and `ssi_take_up.py` import it without behavior changes. -- Passed repository-wide Ruff and `git diff --check`. -- Passed the full `populace-build` suite: 3,335 passed, 85 skipped, with five - pre-existing warnings, in 107.47 seconds. -- Prepared an accurate replacement PR body covering the enumerated contract, - the sole `operator_boundary.py` owner addition, and all three runtime - constant-plumbing files. The live connector update was canceled, so the - replacement will be preserved verbatim in the final handoff rather than - applied through another channel. +- Confirmed a clean starting worktree at `aef1c56`. +- Read the repository guidance and established the #515 donor carve as the + screen's required downstream boundary. +- Started source-level audits of every donor-frame consumer, checkpoint + validation, row-count pins, and existing donor-fact summaries. +- Attempted the requested GitNexus impact workflow; the managed filesystem + denied its global registry write. Its local index also exposed a broad + `build/` ignore mismatch, so the completed impact audit uses direct source + call sites and tests. +- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the + structural rationale and pinned-artifact receipts. +- Added a whole-row screen on grouped raw person `home_mortgage_interest` + after tax-unit assembly, before the #515 carve, with retained-index reset. +- Confirmed no downstream consumer pairs donor rows to the original HDF arrays + or carries a stale donor-length vector; values and weights always originate + from the same screened frame. +- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale + checkpoint regression track the live constant while retaining literal-v1 + corruptions. +- Added regression coverage for the exact grouped boundary, whole-row removal, + retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant. +- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets + 139; populace-data 138 with 1 skip. The directly affected tail-bound suite + adds 12 passes. Ruff format/check and `git diff --check` are clean. +- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line + audit, expected 208,611-row real-artifact effect, verification results, count + sweep, and deliberately untouched surfaces. ## Next -- Write `/private/tmp/583_fix4_handoff.md` with exact receipts and the - unapplied PR-body replacement. -- Restore `PROGRESS.md` byte-for-byte to `origin/main` before final handoff. +- PR #527 review cycle, then merge. After both #525 and #527: rebuild the + base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a + run that holds per `us_critical_targets.py`. +- Root record-level ETL carve stays open on populace#515. From 142e133ca35aedaba9c8de525d2a7550c479b919 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 02:25:34 -0400 Subject: [PATCH 60/94] Round 6: qualified factory aliases and pair-unpacking loops enter scope MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sol round-6 natural-code gaps: (1) module-qualified references to canonical factories (provenance.support_channel_column, plain or named-expression) now register as factory aliases — qualified imports are ordinary code, not obfuscation; (2) tuple-unpacking loops and comprehensions over statically enumerable rows of strings bind each target to its per-position choice set, so declarative (entity, suffix) pair tables resolve by name — with benign pair tables staying clean. Both reviewer repros plus a comprehension variant are committed self-tests. Co-Authored-By: Claude Fable 5 --- .../tests/test_us_spine_blindness.py | 125 ++++++++++++++++-- 1 file changed, 116 insertions(+), 9 deletions(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 99668b42..f2ae3092 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -314,7 +314,16 @@ def _factory_aliases(tree: ast.AST) -> set[str]: ) else: continue - if not isinstance(value, ast.Name) or value.id not in aliases: + if isinstance(value, ast.Name): + if value.id not in aliases: + continue + elif isinstance(value, ast.Attribute): + # provenance.support_channel_column — module-qualified + # references to canonical factories are ordinary imports, + # not obfuscation (sol #583 round 6). + if value.attr not in _SOURCE_SPINE_COLUMN_FACTORIES: + continue + else: continue for target in targets: if isinstance(target, ast.Name) and target.id not in aliases: @@ -1336,6 +1345,43 @@ def visit_NamedExpr(self, node: ast.NamedExpr) -> None: ) self._bind([node.target], node.value, scope_index=scope_index) + def _bind_iteration_rows( + self, + target: ast.AST, + iterable: ast.AST, + ) -> bool: + """Bind tuple-unpacking loop targets over static rows of strings. + + ``for entity, suffix in PAIRS`` with ``PAIRS = (("person", + "support_channel"),)`` binds ``entity``/``suffix`` to their + per-position choice sets — natural declarative loop code, in + scope (sol #583 round 6). + """ + + literal = _static_literal_value(iterable, self.constants) + if literal is _OPAQUE_STATIC_VALUE or not isinstance(literal, (list, tuple)): + return False + rows = tuple(literal) + if not rows or not all( + isinstance(row, (list, tuple)) + and all(isinstance(item, str) for item in row) + for row in rows + ): + return False + width = len(rows[0]) + if any(len(row) != width for row in rows) or not ( + isinstance(target, (ast.Tuple, ast.List)) + and len(target.elts) == width + and all(isinstance(element, ast.Name) for element in target.elts) + ): + return False + for position, element in enumerate(target.elts): + self._bind_iteration_target( + element, + tuple(row[position] for row in rows), + ) + return True + def _bind_iteration_target( self, target: ast.AST, @@ -1512,10 +1558,13 @@ def visit_For(self, node: ast.For) -> None: self._visit_access_target(node.target) before = self._flow_state() values = _static_string_list(node.iter, self.constants) - self._bind_iteration_target( - node.target, - values, - ) + if values is None and self._bind_iteration_rows(node.target, node.iter): + values = ("",) # rows bound per position; body always analyzed + else: + self._bind_iteration_target( + node.target, + values, + ) for statement in node.body: self.visit(statement) body_state = self._flow_state() @@ -1557,10 +1606,15 @@ def _visit_comprehension( self.visit(generator.iter) values = _static_string_list(generator.iter, self.constants) self._visit_access_target(generator.target) - self._bind_iteration_target( - generator.target, - values, - ) + if values is None and self._bind_iteration_rows( + generator.target, generator.iter + ): + pass + else: + self._bind_iteration_target( + generator.target, + values, + ) for condition in generator.ifs: self.visit(condition) if isinstance(node, ast.DictComp): @@ -3034,6 +3088,59 @@ def f(df): assert not _source_spine_accesses(benign_bound) +def test_qualified_factory_aliases_and_pair_loops_are_in_scope(): + """Sol #583 round-6 natural-code gaps: module-qualified factory + aliases and tuple-unpacking loops over static pair containers are + ordinary code, so they are enforced, not boundary.""" + + qualified_alias = """ +import populace.build.us_runtime.support_provenance as provenance + + +def f(): + factory = provenance.support_channel_column + return factory("person") +""" + qualified_named_expr = """ +import populace.build.us_runtime.support_provenance as provenance + + +def f(): + return (factory := provenance.support_channel_column)("person") +""" + pair_loop = """ +PAIRS = (("person", "support_channel"),) + + +def f(): + for entity, suffix in PAIRS: + sink(f"{entity}_{suffix}") +""" + pair_comprehension = """ +PAIRS = (("person", "support_channel"),) + + +def f(): + return [f"{entity}_{suffix}" for entity, suffix in PAIRS] +""" + benign_pair_loop = """ +PAIRS = (("person", "age"),) + + +def f(): + for entity, suffix in PAIRS: + sink(f"{entity}_{suffix}") +""" + for source in ( + qualified_alias, + qualified_named_expr, + pair_loop, + pair_comprehension, + ): + assert _source_spine_accesses(source), source + assert _source_spine_accesses(benign_pair_loop) == () + + def test_guard_treats_wildcards_hidden_args_and_mutations_as_opaque(): """Sol #583 round-3 evasions: str.format and parameter interpolation resolved to a '*' wildcard and read as benign; kwargs-expansion hid From 53fc85c262d36f6fe9aee1207e3af1e1c4bbf026 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 02:42:18 -0400 Subject: [PATCH 61/94] Round 7: module-local scope stated; dict.items and starred rows enter it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sol round-7 edges adjudicated by the module-local principle: static dict.items() iteration and starred/mixed-width row unpacking are module-local ordinary code and now resolve (with the *rest tail bound opaque); imported constant tables and namespace re-export hops are whole-program dataflow, now the docstring's named first boundary with code review as the control, pinned by documented-boundary tests. The Cartesian over-catch on combined multi-value bindings is documented as deliberate conservatism — over-reporting is the safe failure direction for a tripwire — and pinned so a silent behavior change surfaces. Co-Authored-By: Claude Fable 5 --- .../tests/test_us_spine_blindness.py | 152 ++++++++++++++++-- 1 file changed, 136 insertions(+), 16 deletions(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index f2ae3092..df7cab47 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -16,14 +16,23 @@ - contraband guarded-name literals anywhere statically visible in non-owner modules. -Two classes are out of scope by design, and naming them is the honest -boundary. First, deliberately obfuscated construction -- reverse slicing, -``format_map`` over dynamic maps, ``__doc__`` or ``__annotations__`` mining, -container-indexed method aliases, and kin -- is controlled by code review and -the adversarial merge-review process. A scanner that claimed to catch code -written to deceive would be lying. Second, column names materialized purely -from runtime data are controlled by the assembly receipt and runtime -validation. +Analysis is MODULE-LOCAL with single-hop name resolution. Three classes +are out of scope by design, and naming them is the honest boundary. +First, cross-module static dataflow -- constant tables imported from +other modules and namespace re-export or attribute-mutation hops -- +resolves only within the scanned module; whole-program dataflow is code +review's job. Second, deliberately obfuscated construction -- reverse +slicing, ``format_map`` over dynamic maps, ``__doc__`` or +``__annotations__`` mining, container-indexed method aliases, and kin -- +is controlled by code review and the adversarial merge-review process. A +scanner that claimed to catch code written to deceive would be lying. +Third, column names materialized purely from runtime data are controlled +by the assembly receipt and runtime validation. + +Where multi-value loop bindings combine in one template, the guard may +over-report combinations that no single row produces (a Cartesian +over-catch). Over-reporting is the safe failure direction for a +tripwire; a module that trips it restructures its table. """ from __future__ import annotations @@ -1358,7 +1367,21 @@ def _bind_iteration_rows( scope (sol #583 round 6). """ - literal = _static_literal_value(iterable, self.constants) + if ( + isinstance(iterable, ast.Call) + and isinstance(iterable.func, ast.Attribute) + and iterable.func.attr == "items" + and not iterable.args + and not iterable.keywords + ): + # Static dict.items() iteration is module-local and ordinary + # (sol #583 round 7). + mapping = _static_literal_value(iterable.func.value, self.constants) + if not isinstance(mapping, dict): + return False + literal: object = tuple(mapping.items()) + else: + literal = _static_literal_value(iterable, self.constants) if literal is _OPAQUE_STATIC_VALUE or not isinstance(literal, (list, tuple)): return False rows = tuple(literal) @@ -1368,18 +1391,28 @@ def _bind_iteration_rows( for row in rows ): return False - width = len(rows[0]) - if any(len(row) != width for row in rows) or not ( - isinstance(target, (ast.Tuple, ast.List)) - and len(target.elts) == width - and all(isinstance(element, ast.Name) for element in target.elts) - ): + if not isinstance(target, (ast.Tuple, ast.List)): return False - for position, element in enumerate(target.elts): + elements = list(target.elts) + starred = bool(elements) and isinstance(elements[-1], ast.Starred) + leading = elements[:-1] if starred else elements + if not leading or not all(isinstance(element, ast.Name) for element in leading): + return False + prefix = len(leading) + if starred: + # Mixed row widths are fine when a *rest tail absorbs them + # (sol #583 round 7); the tail itself binds opaque. + if any(len(row) < prefix for row in rows): + return False + elif any(len(row) != prefix for row in rows): + return False + for position, element in enumerate(leading): self._bind_iteration_target( element, tuple(row[position] for row in rows), ) + if starred and isinstance(elements[-1].value, ast.Name): + self._bind_iteration_target(elements[-1].value, None) return True def _bind_iteration_target( @@ -3088,6 +3121,93 @@ def f(df): assert not _source_spine_accesses(benign_bound) +def test_dict_items_and_starred_row_iteration_are_in_scope(): + """Sol #583 round-7 module-local edges: static dict.items() and + starred/mixed-width row unpacking are ordinary declarative code.""" + + items_loop = """ +PAIRS = {"person": "support_channel"} + + +def f(): + for entity, suffix in PAIRS.items(): + sink(f"{entity}_{suffix}") +""" + items_comprehension = """ +PAIRS = {"person": "support_channel"} + + +def f(): + return [f"{e}_{sfx}" for e, sfx in PAIRS.items()] +""" + starred_rows = """ +ROWS = (("person", "support_channel"), ("household", "age", "ignored")) + + +def f(): + for entity, suffix, *rest in ROWS: + sink(f"{entity}_{suffix}") +""" + benign_starred = """ +ROWS = (("person", "age"), ("household", "weight", "x")) + + +def f(): + for entity, suffix, *rest in ROWS: + sink(f"{entity}_{suffix}") +""" + for source in (items_loop, items_comprehension, starred_rows): + assert _source_spine_accesses(source), source + assert _source_spine_accesses(benign_starred) == () + + +def test_cartesian_over_catch_is_documented_conservatism(): + """Multi-value bindings combined in one template may report + combinations no single row produces. Over-reporting is the safe + failure direction for a tripwire (module docstring); this pins the + behavior so a silent change surfaces here.""" + + cross_row = """ +ROWS = (("person", "age"), ("metric", "support_channel")) + + +def f(): + for entity, suffix in ROWS: + sink(f"{entity}_{suffix}") +""" + accesses = _source_spine_accesses(cross_row) + assert any("person_support_channel" in access for access in accesses) + + +def test_cross_module_dataflow_is_documented_boundary(): + """Imported constant tables and namespace re-export hops are + module-local-analysis boundaries (docstring class one): the consumer + module scans clean and code review is the stated control.""" + + imported_pairs_consumer = """ +from catalog import PAIRS + + +def f(): + for entity, suffix in PAIRS: + sink(f"{entity}_{suffix}") +""" + namespace_rebound_alias = """ +import populace.build.us_runtime.support_provenance as provenance + +import factories + +factories.channel_factory = provenance.support_channel_column + + +def f(): + factory = factories.channel_factory + return factory("person") +""" + assert _source_spine_accesses(imported_pairs_consumer) == () + assert _source_spine_accesses(namespace_rebound_alias) == () + + def test_qualified_factory_aliases_and_pair_loops_are_in_scope(): """Sol #583 round-6 natural-code gaps: module-qualified factory aliases and tuple-unpacking loops over static pair containers are From 22d7771d0641a74c1918566015dcbef2d7f48b48 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 02:59:52 -0400 Subject: [PATCH 62/94] Round 8: any-position stars and constructed dict entries enter scope MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sol round-8 module-local leaks: (1) a star in a non-tail target position slipped the rows binder — a single star anywhere now absorbs mixed widths, with leading names bound from the row front and trailing names from the row END; (2) static string construction inside dict entries (concatenation, format, case folding) fell to opaque — the literal resolver now finishes with a wildcard-free string-shape fallback, so constructed keys and values resolve like any literal. Both repros, a mid-star comprehension, and a benign dict.items control are committed self-tests. Co-Authored-By: Claude Fable 5 --- .../tests/test_us_spine_blindness.py | 88 ++++++++++++++++--- 1 file changed, 77 insertions(+), 11 deletions(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index df7cab47..41357fc8 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -568,6 +568,12 @@ def _static_literal_value( return dict(zip(keys, values, strict=True)) except TypeError: return _OPAQUE_STATIC_VALUE + shape = _static_string_shape(node, constants) + if shape is not None and _OPAQUE_STRING_PART not in shape: + # Static string construction (concatenation, format, case folding) + # inside container entries resolves like any other literal + # (sol #583 round 8). + return shape return _OPAQUE_STATIC_VALUE @@ -1394,25 +1400,45 @@ def _bind_iteration_rows( if not isinstance(target, (ast.Tuple, ast.List)): return False elements = list(target.elts) - starred = bool(elements) and isinstance(elements[-1], ast.Starred) - leading = elements[:-1] if starred else elements - if not leading or not all(isinstance(element, ast.Name) for element in leading): + star_positions = [ + index + for index, element in enumerate(elements) + if isinstance(element, ast.Starred) + ] + if len(star_positions) > 1: + return False + if star_positions: + # A star ANYWHERE absorbs mixed widths; names before it bind + # from the row front, names after it bind from the row END + # (sol #583 rounds 7-8). The star itself binds opaque. + star = star_positions[0] + leading = elements[:star] + trailing = elements[star + 1 :] + else: + leading = elements + trailing = [] + if not (leading or trailing) or not all( + isinstance(element, ast.Name) for element in (*leading, *trailing) + ): return False - prefix = len(leading) - if starred: - # Mixed row widths are fine when a *rest tail absorbs them - # (sol #583 round 7); the tail itself binds opaque. - if any(len(row) < prefix for row in rows): + needed = len(leading) + len(trailing) + if star_positions: + if any(len(row) < needed for row in rows): return False - elif any(len(row) != prefix for row in rows): + elif any(len(row) != len(leading) for row in rows): return False for position, element in enumerate(leading): self._bind_iteration_target( element, tuple(row[position] for row in rows), ) - if starred and isinstance(elements[-1].value, ast.Name): - self._bind_iteration_target(elements[-1].value, None) + for back, element in enumerate(reversed(trailing), start=1): + self._bind_iteration_target( + element, + tuple(row[-back] for row in rows), + ) + if star_positions and isinstance(elements[star_positions[0]].value, ast.Name): + self._bind_iteration_target(elements[star_positions[0]].value, None) return True def _bind_iteration_target( @@ -3161,6 +3187,46 @@ def f(): assert _source_spine_accesses(benign_starred) == () +def test_mid_star_rows_and_concatenated_dict_entries_are_in_scope(): + """Sol #583 round-8 module-local edges: a star in any single target + position binds trailing names from the row end, and static string + construction inside dict entries resolves like any literal.""" + + mid_star = """ +ROWS = (("person", "x", "support_channel"), ("household", "a", "b", "age")) + + +def f(): + for entity, *rest, suffix in ROWS: + sink(f"{entity}_{suffix}") +""" + mid_star_comprehension = """ +ROWS = (("person", "x", "support_channel"),) + + +def f(): + return [f"{e}_{sfx}" for e, *rest, sfx in ROWS] +""" + concat_items = """ +def f(): + return [ + f"{entity}_{suffix}" + for entity, suffix in {"per" + "son": "support_" + "channel"}.items() + ] +""" + benign_items = """ +PAIRS = {"person": "age"} + + +def f(): + for entity, suffix in PAIRS.items(): + sink(f"{entity}_{suffix}") +""" + for source in (mid_star, mid_star_comprehension, concat_items): + assert _source_spine_accesses(source), source + assert _source_spine_accesses(benign_items) == () + + def test_cartesian_over_catch_is_documented_conservatism(): """Multi-value bindings combined in one template may report combinations no single row produces. Over-reporting is the safe From 9e9acf58af0b3c1ed6397a10ef164954cc490d42 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 03:20:56 -0400 Subject: [PATCH 63/94] Round 9: unpropagatable geometry over guarded fragments fails closed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The structural end of the geometry ladder: when an iterated container statically carries fragments of guarded column names (by expression or by resolved value) and the rows binder cannot propagate to the target geometry, the LOOP records a fail-closed violation instead of binding silently opaque — so nested star payloads, rows-of-rows, dict() constructors, and any future shape are born caught. Two precision controls keep production clean: the fragment probes apply only to genuinely static containers (dynamic iterables that merely mention strings, like mapping.get('person', {}).values(), never trip), and the binder gained per-column partial propagation so mixed rows of static labels beside dynamic objects — the congressional-district vintage pattern — propagate instead of refusing. Sol's three round-9 repros, the dynamic-mention control, and a fragment-free nested control are committed self-tests. Co-Authored-By: Claude Fable 5 --- .../tests/test_us_spine_blindness.py | 200 +++++++++++++++++- 1 file changed, 193 insertions(+), 7 deletions(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 41357fc8..6a12fb00 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -1130,6 +1130,115 @@ def _scope_assignment_counts( return counter.counts +def _static_structure( + node: ast.AST, constants: list[dict[str, object]] +) -> tuple | None: + """Resolve a list/tuple container STRUCTURALLY, tolerating opaque + leaves (kept as the opaque sentinel). Lets the rows binder propagate + mixed rows — static labels beside dynamic objects — per column.""" + + if isinstance(node, ast.Name): + for scope in reversed(constants): + if node.id in scope: + value = scope[node.id] + if isinstance(value, (list, tuple)): + return tuple(value) + return None + return None + if isinstance(node, (ast.List, ast.Tuple)): + resolved = [] + for element in node.elts: + if isinstance(element, (ast.List, ast.Tuple)): + inner = _static_structure(element, constants) + resolved.append(inner if inner is not None else _OPAQUE_STATIC_VALUE) + else: + value = _static_literal_value(element, constants) + resolved.append(value) + return tuple(resolved) + return None + + +def _static_container_expression(node: ast.AST) -> ast.AST | None: + """The literal-container core of an iterable expression, if any. + + Recognizes literal containers directly, ``.items()``, and + ``dict()`` — the forms whose string leaves are + guaranteed container CONTENT. Dynamic expressions that merely mention + strings (``mapping.get("person", {}).values()``) return None so key + lookups never trip the fragment rule.""" + + if isinstance(node, (ast.List, ast.Tuple, ast.Set, ast.Dict)): + return node + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and node.func.attr == "items" + and not node.args + and not node.keywords + ): + return _static_container_expression(node.func.value) + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "dict" + and len(node.args) == 1 + and not node.keywords + ): + return _static_container_expression(node.args[0]) + return None + + +def _iterable_carries_guarded_fragments(node: ast.AST) -> bool: + """True when a literal-container iterable holds string constants that + assemble into a guarded column name. Applies only to genuinely static + container expressions (see _static_container_expression) so dynamic + iterables that merely mention strings never trip it. Fragments shorter + than four characters are ignored.""" + + container = _static_container_expression(node) + if container is None: + return False + fragments = [ + constant.value + for constant in ast.walk(container) + if isinstance(constant, ast.Constant) + and isinstance(constant.value, str) + and len(constant.value) >= 4 + ] + return any( + fragment in column + for fragment in fragments + for column in _OPERATOR_SOURCE_COLUMNS + ) + + +def _resolved_value_carries_guarded_fragments( + node: ast.AST, constants: list[dict[str, object]] +) -> bool: + """The bound-value counterpart of the syntactic probe: when the + iterable is a name (or expression) resolving to a static structure, + walk that structure's string leaves for guarded fragments.""" + + value = _static_literal_value(node, constants) + if value is _OPAQUE_STATIC_VALUE: + return False + leaves: list[str] = [] + stack = [value] + while stack: + item = stack.pop() + if isinstance(item, str): + if len(item) >= 4: + leaves.append(item) + elif isinstance(item, (list, tuple, set, frozenset)): + stack.extend(item) + elif isinstance(item, dict): + stack.extend(item.keys()) + stack.extend(item.values()) + return any( + fragment in column for fragment in leaves for column in _OPERATOR_SOURCE_COLUMNS + ) + + class _SourceReadVisitor(ast.NodeVisitor): def __init__( self, @@ -1388,14 +1497,16 @@ def _bind_iteration_rows( literal: object = tuple(mapping.items()) else: literal = _static_literal_value(iterable, self.constants) + if literal is _OPAQUE_STATIC_VALUE or not isinstance( + literal, (list, tuple) + ): + structure = _static_structure(iterable, self.constants) + if structure is not None: + literal = structure if literal is _OPAQUE_STATIC_VALUE or not isinstance(literal, (list, tuple)): return False rows = tuple(literal) - if not rows or not all( - isinstance(row, (list, tuple)) - and all(isinstance(item, str) for item in row) - for row in rows - ): + if not rows or not all(isinstance(row, (list, tuple)) for row in rows): return False if not isinstance(target, (ast.Tuple, ast.List)): return False @@ -1427,15 +1538,19 @@ def _bind_iteration_rows( return False elif any(len(row) != len(leading) for row in rows): return False + + def _column_choices(values: tuple) -> tuple[str, ...] | None: + return values if all(isinstance(value, str) for value in values) else None + for position, element in enumerate(leading): self._bind_iteration_target( element, - tuple(row[position] for row in rows), + _column_choices(tuple(row[position] for row in rows)), ) for back, element in enumerate(reversed(trailing), start=1): self._bind_iteration_target( element, - tuple(row[-back] for row in rows), + _column_choices(tuple(row[-back] for row in rows)), ) if star_positions and isinstance(elements[star_positions[0]].value, ast.Name): self._bind_iteration_target(elements[star_positions[0]].value, None) @@ -1620,6 +1735,20 @@ def visit_For(self, node: ast.For) -> None: if values is None and self._bind_iteration_rows(node.target, node.iter): values = ("",) # rows bound per position; body always analyzed else: + if values is None and ( + _iterable_carries_guarded_fragments(node.iter) + or _resolved_value_carries_guarded_fragments(node.iter, self.constants) + ): + # The container statically carries guarded-name fragments + # but the binder cannot propagate them to this target + # geometry — the loop itself fails closed rather than + # binding silently opaque (sol #583 round 9). + self._record( + node, + "iteration over a static container carrying guarded-name " + "fragments with unpropagatable target geometry " + "(fail-closed)", + ) self._bind_iteration_target( node.target, values, @@ -1670,6 +1799,18 @@ def _visit_comprehension( ): pass else: + if values is None and ( + _iterable_carries_guarded_fragments(generator.iter) + or _resolved_value_carries_guarded_fragments( + generator.iter, self.constants + ) + ): + self._record( + generator.iter, + "iteration over a static container carrying " + "guarded-name fragments with unpropagatable target " + "geometry (fail-closed)", + ) self._bind_iteration_target( generator.target, values, @@ -3187,6 +3328,51 @@ def f(): assert _source_spine_accesses(benign_starred) == () +def test_unpropagatable_geometry_over_guarded_fragments_fails_closed(): + """Sol #583 round-9: any target/container geometry the binder cannot + propagate fails closed AT THE LOOP when the container statically + carries guarded-name fragments — ending geometry-by-geometry chasing. + Dynamic iterables that merely mention strings (key lookups) and + fragment-free static tables stay clean.""" + + nested_star_payload = """ +ROWS = (("person", "support", "channel"),) + + +def f(df): + return [df.filter(items=[f"{e}_{m}_{sfx}"]) for *(e, m), sfx in ROWS] +""" + rows_of_rows = """ +ROWS = ((("person", "support_channel"), "metadata"),) + + +def f(): + for (entity, suffix), metadata in ROWS: + sink(f"{entity}_{suffix}") +""" + dict_constructor = """ +def f(): + for entity, suffix in dict([("person", "support_channel")]).items(): + sink(f"{entity}_{suffix}") +""" + dynamic_mentioning_strings = """ +def f(target_families): + return {t for ts in target_families.get("person", {}).values() for t in ts} +""" + fragment_free_nested = """ +ROWS = ((("state", "fips"), "meta"),) + + +def f(): + for (a, b), meta in ROWS: + sink(f"{a}_{b}") +""" + for source in (nested_star_payload, rows_of_rows, dict_constructor): + assert _source_spine_accesses(source), source + assert _source_spine_accesses(dynamic_mentioning_strings) == () + assert _source_spine_accesses(fragment_free_nested) == () + + def test_mid_star_rows_and_concatenated_dict_entries_are_in_scope(): """Sol #583 round-8 module-local edges: a star in any single target position binds trailing names from the row end, and static string From 87a8fc89330dbbcc61bff178144d5130d93a305e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 03:37:42 -0400 Subject: [PATCH 64/94] Start PR 583 round 10 fix journal --- PROGRESS.md | 57 ++++++++++++++++++----------------------------------- 1 file changed, 19 insertions(+), 38 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index f3158fdd..a0165d52 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,47 +2,28 @@ ## State -Populace #516 whole-row donor outlier screen is complete on -`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515 -interim carve merged as #525). The `puf_tax_detail` donor now drops tax units -whose grouped raw mortgage interest reaches $10M before the #515 carve -(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T -of phantom mortgage-interest mass), with the checkpoint schema bumped to v3 -so post-carve pre-screen checkpoints rebuild. +PR #583 round-10 fix lane is in progress on +`multispine-pool-build-578` from `9e9acf5`. The review HOLD identifies four +mechanical closure gaps in the spine-blindness structural guard: nested-star +payload poisoning, non-name loop targets, static-dict `.values()` iteration, +and partial structures propagated through bindings. ## Done -- Confirmed a clean starting worktree at `aef1c56`. -- Read the repository guidance and established the #515 donor carve as the - screen's required downstream boundary. -- Started source-level audits of every donor-frame consumer, checkpoint - validation, row-count pins, and existing donor-fact summaries. -- Attempted the requested GitNexus impact workflow; the managed filesystem - denied its global registry write. Its local index also exposed a broad - `build/` ignore mismatch, so the completed impact audit uses direct source - call sites and tests. -- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the - structural rationale and pinned-artifact receipts. -- Added a whole-row screen on grouped raw person `home_mortgage_interest` - after tax-unit assembly, before the #515 carve, with retained-index reset. -- Confirmed no downstream consumer pairs donor rows to the original HDF arrays - or carries a stale donor-length vector; values and weights always originate - from the same screened frame. -- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale - checkpoint regression track the live constant while retaining literal-v1 - corruptions. -- Added regression coverage for the exact grouped boundary, whole-row removal, - retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant. -- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets - 139; populace-data 138 with 1 skip. The directly affected tail-bound suite - adds 12 passes. Ruff format/check and `git diff --check` are clean. -- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line - audit, expected 208,611-row real-artifact effect, verification results, count - sweep, and deliberately untouched surfaces. +- Confirmed the requested branch, clean worktree, and exact starting commit. +- Read `CLAUDE.md`, the GitNexus debugging skill, and the authenticated + round-10 review log. +- Recorded the required implementation, regression, documentation, precision, + suite, lint, graph-cleanliness, and no-push constraints. ## Next -- PR #527 review cycle, then merge. After both #525 and #527: rebuild the - base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a - run that holds per `us_critical_targets.py`. -- Root record-level ETL carve stays open on populace#515. +- Reproduce and trace all four bypasses through the current binder and loop + fallback. +- Commit the reviewer's repros as self-tests, implement the four structural + fixes, and make the guard docstring match the final mechanics. +- Run the guard file, full `populace-build` suite, repository ruff, and the + requested `acs_transfer` plus `congressional_district_vintage` cleanliness + checks. +- Restore `PROGRESS.md` exactly to `origin/main`, commit all coherent steps + locally without pushing, and write `/private/tmp/583_fix5_handoff.md`. From 7218dae4ae82fd40a635b82b7188d4a557f45b4f Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 03:46:44 -0400 Subject: [PATCH 65/94] Add round 10 spine guard regression repros --- PROGRESS.md | 16 +++- .../tests/test_us_spine_blindness.py | 91 ++++++++++++++++++- 2 files changed, 101 insertions(+), 6 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index a0165d52..1633f648 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -15,13 +15,21 @@ and partial structures propagated through bindings. round-10 review log. - Recorded the required implementation, regression, documentation, precision, suite, lint, graph-cleanliness, and no-push constraints. +- Confirmed the GitNexus CLI is installed but the repository has no index; a + non-augmenting index attempt was blocked by the managed global-registry + write, and its generated untracked local index was removed. +- Reproduced all six authenticated forms as silent at the starting guard: + stale-outer nested star, attribute target, subscript target, literal + `.values()`, constructor `.values()`, and a bound mixed row. +- Added those exact inputs as self-tests, with loop-diagnostic assertions for + unpropagatable geometry and exact-column assertions for successful + per-column propagation. The focused red run fails all three test groups as + expected before the implementation. ## Next -- Reproduce and trace all four bypasses through the current binder and loop - fallback. -- Commit the reviewer's repros as self-tests, implement the four structural - fixes, and make the guard docstring match the final mechanics. +- Implement the four structural fixes and make the guard docstrings match the + final mechanics. - Run the guard file, full `populace-build` suite, repository ruff, and the requested `acs_transfer` plus `congressional_district_vintage` cleanliness checks. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 6a12fb00..ffe87830 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -3336,11 +3336,16 @@ def test_unpropagatable_geometry_over_guarded_fragments_fails_closed(): fragment-free static tables stay clean.""" nested_star_payload = """ +entity = "age" +middle = "income" ROWS = (("person", "support", "channel"),) def f(df): - return [df.filter(items=[f"{e}_{m}_{sfx}"]) for *(e, m), sfx in ROWS] + return [ + df.filter(items=[f"{entity}_{middle}_{suffix}"]) + for *(entity, middle), suffix in ROWS + ] """ rows_of_rows = """ ROWS = ((("person", "support_channel"), "metadata"),) @@ -3367,12 +3372,94 @@ def f(): for (a, b), meta in ROWS: sink(f"{a}_{b}") """ - for source in (nested_star_payload, rows_of_rows, dict_constructor): + nested_star_accesses = _source_spine_accesses(nested_star_payload) + assert any( + "unpropagatable target geometry" in access + for access in nested_star_accesses + ) + for source in (rows_of_rows, dict_constructor): assert _source_spine_accesses(source), source assert _source_spine_accesses(dynamic_mentioning_strings) == () assert _source_spine_accesses(fragment_free_nested) == () +def test_non_name_targets_over_guarded_static_rows_fail_closed_at_loop(): + """Sol #583 round-10: attribute and subscript stores cannot receive + abstract row choices, so a fragment-bearing static iterable is rejected + at the loop even when its flattened strings are statically enumerable.""" + + attribute_target = """class Row: + pass + +row = Row() +seen = [] +for row.entity, row.suffix in (("person", "support_channel"),): + seen.append(f"{row.entity}_{row.suffix}") +""" + subscript_target = """row = [None, None] +seen = [] +for row[0], row[1] in (("person", "support_channel"),): + seen.append(f"{row[0]}_{row[1]}") +""" + for source in (attribute_target, subscript_target): + accesses = _source_spine_accesses(source) + assert any( + "unpropagatable target geometry" in access for access in accesses + ), source + + +def test_static_dict_values_and_bound_partial_rows_propagate_per_column(): + """Sol #583 round-10: static mapping values and partial structures + retained through a name binding propagate their resolvable columns.""" + + literal_values = """ +def f(): + for entity, middle, suffix in { + "row": ("person", "support", "channel") + }.values(): + sink(f"{entity}_{middle}_{suffix}") +""" + constructor_values = """ +def f(): + for entity, middle, suffix in dict( + [("row", ("person", "support", "channel"))] + ).values(): + sink(f"{entity}_{middle}_{suffix}") +""" + bound_mixed_rows = """ +def f(dynamic_object): + rows = (("person", dynamic_object, "support_channel"),) + for entity, obj, suffix in rows: + sink(f"{entity}_{suffix}") +""" + for source in (literal_values, constructor_values, bound_mixed_rows): + accesses = _source_spine_accesses(source) + assert any("person_support_channel" in access for access in accesses), source + assert not any( + "unpropagatable target geometry" in access for access in accesses + ), source + + literal_values_bad_geometry = """ +def f(row): + for row.entity, row.middle, row.suffix in { + "row": ("person", "support", "channel") + }.values(): + sink(f"{row.entity}_{row.middle}_{row.suffix}") +""" + constructor_values_bad_geometry = """ +def f(row): + for row.entity, row.middle, row.suffix in dict( + [("row", ("person", "support", "channel"))] + ).values(): + sink(f"{row.entity}_{row.middle}_{row.suffix}") +""" + for source in (literal_values_bad_geometry, constructor_values_bad_geometry): + accesses = _source_spine_accesses(source) + assert any( + "unpropagatable target geometry" in access for access in accesses + ), source + + def test_mid_star_rows_and_concatenated_dict_entries_are_in_scope(): """Sol #583 round-8 module-local edges: a star in any single target position binds trailing names from the row end, and static string From 6c8b38fd705b98b40089ed5dc4fda721e1440fa1 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 03:54:11 -0400 Subject: [PATCH 66/94] Close static iteration propagation gaps --- PROGRESS.md | 17 +- .../tests/test_us_spine_blindness.py | 295 +++++++++++++----- 2 files changed, 225 insertions(+), 87 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 1633f648..fbd3f88c 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -25,11 +25,24 @@ and partial structures propagated through bindings. unpropagatable geometry and exact-column assertions for successful per-column propagation. The focused red run fails all three test groups as expected before the implementation. +- Reworked static-row binding to report recognized and fully propagated + status separately. Nested star payload names are all poisoned, partial and + refused targets now enter the loop fallback regardless of flattened-string + resolution, and fragment state is captured before target bindings mutate + the scope. +- Added static `.items()`/`.values()` iteration resolution for literal, + bound, and supported `dict(...)` mappings, with value-only syntactic + fragment probing so guarded-looking mapping keys do not taint benign + `.values()` loops. +- Preserved partial list/tuple structures at assignment bind time, allowing + bound mixed rows to propagate their static columns exactly. +- The 121-case guard and focused 22-case benign/runtime/graph battery pass. + Direct scans report zero findings for both `acs_transfer.py` and + `congressional_district_vintage.py`; file-scoped ruff and formatting pass. ## Next -- Implement the four structural fixes and make the guard docstrings match the - final mechanics. +- Make the guard docstrings match the final mechanics exactly. - Run the guard file, full `populace-build` suite, repository ruff, and the requested `acs_transfer` plus `congressional_district_vintage` cleanliness checks. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index ffe87830..fa3524f4 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -1158,24 +1158,82 @@ def _static_structure( return None +def _static_dict_value( + node: ast.AST, + constants: list[dict[str, object]], +) -> dict | None: + """Resolve a literal/bound dict or one supported ``dict(iterable)``.""" + + mapping = _static_literal_value(node, constants) + if isinstance(mapping, dict): + return mapping + if not ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "dict" + and len(node.args) == 1 + and not node.keywords + ): + return None + entries = _static_literal_value(node.args[0], constants) + if entries is _OPAQUE_STATIC_VALUE: + structure = _static_structure(node.args[0], constants) + if structure is None: + return None + entries = structure + try: + return dict(entries) + except (TypeError, ValueError): + return None + + +def _static_dict_values_expression(node: ast.AST) -> ast.AST | None: + """Return only the syntactic value payload of a supported static dict.""" + + if isinstance(node, ast.Dict): + if any(key is None for key in node.keys): + return None + return ast.Tuple(elts=list(node.values), ctx=ast.Load()) + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "dict" + and len(node.args) == 1 + and not node.keywords + ): + argument = node.args[0] + if isinstance(argument, ast.Dict): + return _static_dict_values_expression(argument) + if isinstance(argument, (ast.List, ast.Tuple, ast.Set)): + values: list[ast.AST] = [] + for entry in argument.elts: + if not isinstance(entry, (ast.List, ast.Tuple)) or len(entry.elts) != 2: + return None + values.append(entry.elts[1]) + return ast.Tuple(elts=values, ctx=ast.Load()) + return None + + def _static_container_expression(node: ast.AST) -> ast.AST | None: """The literal-container core of an iterable expression, if any. - Recognizes literal containers directly, ``.items()``, and - ``dict()`` — the forms whose string leaves are - guaranteed container CONTENT. Dynamic expressions that merely mention - strings (``mapping.get("person", {}).values()``) return None so key - lookups never trip the fragment rule.""" + Recognizes literal containers directly, static-dict ``.items()`` and + ``.values()``, and ``dict()`` — the forms whose string + leaves are guaranteed iterated container content. Dynamic expressions + that merely mention strings (``mapping.get("person", {}).values()``) + return None so key lookups never trip the fragment rule.""" if isinstance(node, (ast.List, ast.Tuple, ast.Set, ast.Dict)): return node if ( isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) - and node.func.attr == "items" + and node.func.attr in {"items", "values"} and not node.args and not node.keywords ): + if node.func.attr == "values": + return _static_dict_values_expression(node.func.value) return _static_container_expression(node.func.value) if ( isinstance(node, ast.Call) @@ -1188,6 +1246,30 @@ def _static_container_expression(node: ast.AST) -> ast.AST | None: return None +def _static_iteration_value( + node: ast.AST, + constants: list[dict[str, object]], +) -> object: + """Resolve the supported static value iterated by a loop binder.""" + + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and node.func.attr in {"items", "values"} + and not node.args + and not node.keywords + ): + mapping = _static_dict_value(node.func.value, constants) + if mapping is None: + return _OPAQUE_STATIC_VALUE + return ( + tuple(mapping.items()) + if node.func.attr == "items" + else tuple(mapping.values()) + ) + return _static_literal_value(node, constants) + + def _iterable_carries_guarded_fragments(node: ast.AST) -> bool: """True when a literal-container iterable holds string constants that assemble into a guarded column name. Applies only to genuinely static @@ -1219,7 +1301,7 @@ def _resolved_value_carries_guarded_fragments( iterable is a name (or expression) resolving to a static structure, walk that structure's string leaves for guarded fragments.""" - value = _static_literal_value(node, constants) + value = _static_iteration_value(node, constants) if value is _OPAQUE_STATIC_VALUE: return False leaves: list[str] = [] @@ -1362,6 +1444,8 @@ def _bind_name( literal = _static_literal_value(value, self.constants) if literal is not _OPAQUE_STATIC_VALUE: constant = literal + if constant is None: + constant = _static_structure(value, self.constants) if constant is None: constant = _static_string_list(value, self.constants) method_alias = self._method_alias(value) @@ -1473,43 +1557,30 @@ def _bind_iteration_rows( self, target: ast.AST, iterable: ast.AST, - ) -> bool: - """Bind tuple-unpacking loop targets over static rows of strings. + ) -> tuple[bool, bool]: + """Bind tuple-unpacking targets over supported static rows. ``for entity, suffix in PAIRS`` with ``PAIRS = (("person", "support_channel"),)`` binds ``entity``/``suffix`` to their per-position choice sets — natural declarative loop code, in - scope (sol #583 round 6). + scope (sol #583 round 6). The result reports whether nonempty + static rows were recognized and whether every target position was + fully propagated. Resolvable string columns remain exact; dynamic + columns and every name in a star payload bind opaque. """ - if ( - isinstance(iterable, ast.Call) - and isinstance(iterable.func, ast.Attribute) - and iterable.func.attr == "items" - and not iterable.args - and not iterable.keywords - ): - # Static dict.items() iteration is module-local and ordinary - # (sol #583 round 7). - mapping = _static_literal_value(iterable.func.value, self.constants) - if not isinstance(mapping, dict): - return False - literal: object = tuple(mapping.items()) - else: - literal = _static_literal_value(iterable, self.constants) - if literal is _OPAQUE_STATIC_VALUE or not isinstance( - literal, (list, tuple) - ): - structure = _static_structure(iterable, self.constants) - if structure is not None: - literal = structure + literal = _static_iteration_value(iterable, self.constants) if literal is _OPAQUE_STATIC_VALUE or not isinstance(literal, (list, tuple)): - return False + structure = _static_structure(iterable, self.constants) + if structure is not None: + literal = structure + if literal is _OPAQUE_STATIC_VALUE or not isinstance(literal, (list, tuple)): + return False, False rows = tuple(literal) if not rows or not all(isinstance(row, (list, tuple)) for row in rows): - return False + return False, False if not isinstance(target, (ast.Tuple, ast.List)): - return False + return True, False elements = list(target.elts) star_positions = [ index @@ -1517,11 +1588,12 @@ def _bind_iteration_rows( if isinstance(element, ast.Starred) ] if len(star_positions) > 1: - return False + return True, False if star_positions: # A star ANYWHERE absorbs mixed widths; names before it bind # from the row front, names after it bind from the row END - # (sol #583 rounds 7-8). The star itself binds opaque. + # (sol #583 rounds 7-8). Every stored name anywhere in the star + # payload binds opaque (sol #583 round 10). star = star_positions[0] leading = elements[:star] trailing = elements[star + 1 :] @@ -1531,13 +1603,13 @@ def _bind_iteration_rows( if not (leading or trailing) or not all( isinstance(element, ast.Name) for element in (*leading, *trailing) ): - return False + return True, False needed = len(leading) + len(trailing) if star_positions: if any(len(row) < needed for row in rows): - return False + return True, False elif any(len(row) != len(leading) for row in rows): - return False + return True, False def _column_choices(values: tuple) -> tuple[str, ...] | None: return values if all(isinstance(value, str) for value in values) else None @@ -1552,15 +1624,21 @@ def _column_choices(values: tuple) -> tuple[str, ...] | None: element, _column_choices(tuple(row[-back] for row in rows)), ) - if star_positions and isinstance(elements[star_positions[0]].value, ast.Name): - self._bind_iteration_target(elements[star_positions[0]].value, None) - return True + if not star_positions: + return True, True + star_payload = elements[star_positions[0]].value + for descendant in ast.walk(star_payload): + if isinstance(descendant, ast.Name) and isinstance( + descendant.ctx, ast.Store + ): + self._bind_iteration_target(descendant, None) + return True, isinstance(star_payload, ast.Name) def _bind_iteration_target( self, target: ast.AST, values: tuple[str, ...] | None, - ) -> None: + ) -> bool: for name in _assigned_names(target): self.bindings[-1][name] = None self.constants[-1][name] = ( @@ -1573,6 +1651,18 @@ def _bind_iteration_target( ) if self.scope_kinds[-1] == "comprehension": self.assignment_counts[-1][name] = 1 + return self._iteration_target_is_plain_names(target) + + @staticmethod + def _iteration_target_is_plain_names(target: ast.AST) -> bool: + if isinstance(target, ast.Name): + return True + if isinstance(target, (ast.List, ast.Tuple)): + return bool(target.elts) and all( + _SourceReadVisitor._iteration_target_is_plain_names(element) + for element in target.elts + ) + return False def _flow_state( self, @@ -1732,27 +1822,31 @@ def visit_For(self, node: ast.For) -> None: self._visit_access_target(node.target) before = self._flow_state() values = _static_string_list(node.iter, self.constants) - if values is None and self._bind_iteration_rows(node.target, node.iter): + carries_guarded_fragments = _iterable_carries_guarded_fragments( + node.iter + ) or _resolved_value_carries_guarded_fragments(node.iter, self.constants) + rows_recognized, fully_propagated = self._bind_iteration_rows( + node.target, + node.iter, + ) + if rows_recognized: values = ("",) # rows bound per position; body always analyzed else: - if values is None and ( - _iterable_carries_guarded_fragments(node.iter) - or _resolved_value_carries_guarded_fragments(node.iter, self.constants) - ): - # The container statically carries guarded-name fragments - # but the binder cannot propagate them to this target - # geometry — the loop itself fails closed rather than - # binding silently opaque (sol #583 round 9). - self._record( - node, - "iteration over a static container carrying guarded-name " - "fragments with unpropagatable target geometry " - "(fail-closed)", - ) - self._bind_iteration_target( + target_propagated = self._bind_iteration_target( node.target, values, ) + fully_propagated = values is not None and target_propagated + if not fully_propagated and carries_guarded_fragments: + # Any refused or partial static binding over guarded fragments + # fails closed at the loop, independently of scalar-string + # enumeration (sol #583 rounds 9-10). + self._record( + node, + "iteration over a static container carrying guarded-name " + "fragments with unpropagatable target geometry " + "(fail-closed)", + ) for statement in node.body: self.visit(statement) body_state = self._flow_state() @@ -1794,27 +1888,29 @@ def _visit_comprehension( self.visit(generator.iter) values = _static_string_list(generator.iter, self.constants) self._visit_access_target(generator.target) - if values is None and self._bind_iteration_rows( - generator.target, generator.iter - ): - pass - else: - if values is None and ( - _iterable_carries_guarded_fragments(generator.iter) - or _resolved_value_carries_guarded_fragments( - generator.iter, self.constants - ) - ): - self._record( - generator.iter, - "iteration over a static container carrying " - "guarded-name fragments with unpropagatable target " - "geometry (fail-closed)", - ) - self._bind_iteration_target( + carries_guarded_fragments = _iterable_carries_guarded_fragments( + generator.iter + ) or _resolved_value_carries_guarded_fragments( + generator.iter, + self.constants, + ) + rows_recognized, fully_propagated = self._bind_iteration_rows( + generator.target, + generator.iter, + ) + if not rows_recognized: + target_propagated = self._bind_iteration_target( generator.target, values, ) + fully_propagated = values is not None and target_propagated + if not fully_propagated and carries_guarded_fragments: + self._record( + generator.iter, + "iteration over a static container carrying " + "guarded-name fragments with unpropagatable target " + "geometry (fail-closed)", + ) for condition in generator.ifs: self.visit(condition) if isinstance(node, ast.DictComp): @@ -3374,8 +3470,7 @@ def f(): """ nested_star_accesses = _source_spine_accesses(nested_star_payload) assert any( - "unpropagatable target geometry" in access - for access in nested_star_accesses + "unpropagatable target geometry" in access for access in nested_star_accesses ) for source in (rows_of_rows, dict_constructor): assert _source_spine_accesses(source), source @@ -3403,9 +3498,9 @@ def test_non_name_targets_over_guarded_static_rows_fail_closed_at_loop(): """ for source in (attribute_target, subscript_target): accesses = _source_spine_accesses(source) - assert any( - "unpropagatable target geometry" in access for access in accesses - ), source + assert any("unpropagatable target geometry" in access for access in accesses), ( + source + ) def test_static_dict_values_and_bound_partial_rows_propagate_per_column(): @@ -3455,9 +3550,39 @@ def f(row): """ for source in (literal_values_bad_geometry, constructor_values_bad_geometry): accesses = _source_spine_accesses(source) - assert any( - "unpropagatable target geometry" in access for access in accesses - ), source + assert any("unpropagatable target geometry" in access for access in accesses), ( + source + ) + + bound_values_bad_geometry = """ +ROWS = {"row": ("person", "support", "channel")} + + +def f(row): + for row.entity, row.middle, row.suffix in ROWS.values(): + sink(f"{row.entity}_{row.middle}_{row.suffix}") +""" + assert any( + "unpropagatable target geometry" in access + for access in _source_spine_accesses(bound_values_bad_geometry) + ) + + key_only_fragment_controls = ( + """ +def f(row): + for row.first, row.second in {"person": ("state", "fips")}.values(): + sink(f"{row.first}_{row.second}") +""", + """ +def f(row): + for row.first, row.second in dict( + [("person", ("state", "fips"))] + ).values(): + sink(f"{row.first}_{row.second}") +""", + ) + for source in key_only_fragment_controls: + assert _source_spine_accesses(source) == (), source def test_mid_star_rows_and_concatenated_dict_entries_are_in_scope(): From 2b9b54cc971b9e0bce10c457bce0dd67ce3a3196 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 03:55:50 -0400 Subject: [PATCH 67/94] Align spine guard contract with iteration mechanics --- PROGRESS.md | 5 +- .../tests/test_us_spine_blindness.py | 56 +++++++++++-------- 2 files changed, 38 insertions(+), 23 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index fbd3f88c..44175774 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -39,10 +39,13 @@ and partial structures propagated through bindings. - The 121-case guard and focused 22-case benign/runtime/graph battery pass. Direct scans report zero findings for both `acs_transfer.py` and `congressional_district_vintage.py`; file-scoped ruff and formatting pass. +- Completed the docstring truth pass: the module contract now names the + supported scalar, structural-row, and static dict-view forms; helper + docstrings distinguish exact propagation, deliberate opacity, and partial + geometry; the round-9 claim now states the actual iteration-site fallback. ## Next -- Make the guard docstrings match the final mechanics exactly. - Run the guard file, full `populace-build` suite, repository ruff, and the requested `acs_transfer` plus `congressional_district_vintage` cleanliness checks. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index fa3524f4..c5d002b9 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -12,7 +12,11 @@ ``str.format`` with full field syntax, ``%`` formatting, ``str * int``, static ``.replace`` chains, and the static-receiver case methods ``lower``, ``upper``, ``casefold``, ``title``, and ``capitalize``; -- loop and comprehension propagation over static iterables; and +- loop and comprehension propagation for supported literal or bound string + choices, structural list/tuple rows, and static-dict ``.items()`` and + ``.values()`` views (including supported ``dict(iterable)`` receivers); + refused or partial binding over a fragment-bearing supported static + container fails closed at the iteration site; and - contraband guarded-name literals anywhere statically visible in non-owner modules. @@ -710,7 +714,7 @@ def _static_percent_operand( def _static_string_list( node: ast.AST, constants: list[dict[str, object]] ) -> tuple[str, ...] | None: - """Resolve any statically enumerable string iterable.""" + """Resolve the supported literal/bound string-choice iterable forms.""" if isinstance(node, ast.Name): for scope in reversed(constants): @@ -1133,9 +1137,11 @@ def _scope_assignment_counts( def _static_structure( node: ast.AST, constants: list[dict[str, object]] ) -> tuple | None: - """Resolve a list/tuple container STRUCTURALLY, tolerating opaque - leaves (kept as the opaque sentinel). Lets the rows binder propagate - mixed rows — static labels beside dynamic objects — per column.""" + """Resolve an inline or bound list/tuple structure with opaque leaves. + + Preserving the sentinel through name bindings lets the rows binder + propagate static columns beside dynamic objects per position. + """ if isinstance(node, ast.Name): for scope in reversed(constants): @@ -1218,10 +1224,11 @@ def _static_container_expression(node: ast.AST) -> ast.AST | None: """The literal-container core of an iterable expression, if any. Recognizes literal containers directly, static-dict ``.items()`` and - ``.values()``, and ``dict()`` — the forms whose string - leaves are guaranteed iterated container content. Dynamic expressions - that merely mention strings (``mapping.get("person", {}).values()``) - return None so key lookups never trip the fragment rule.""" + ``.values()``, and ``dict()``. Their literal content is + eligible for the conservative fragment fallback; ``.values()`` narrows + the core to values rather than mapping keys. Dynamic expressions that + merely mention strings (``mapping.get("person", {}).values()``) return + None so key lookups never trip the fragment rule.""" if isinstance(node, (ast.List, ast.Tuple, ast.Set, ast.Dict)): return node @@ -1297,9 +1304,11 @@ def _iterable_carries_guarded_fragments(node: ast.AST) -> bool: def _resolved_value_carries_guarded_fragments( node: ast.AST, constants: list[dict[str, object]] ) -> bool: - """The bound-value counterpart of the syntactic probe: when the - iterable is a name (or expression) resolving to a static structure, - walk that structure's string leaves for guarded fragments.""" + """Bound-value counterpart of the syntactic fragment probe. + + Names, structures, and supported static-dict views resolve to the value + actually iterated; its string leaves are checked for guarded fragments. + """ value = _static_iteration_value(node, constants) if value is _OPAQUE_STATIC_VALUE: @@ -1563,10 +1572,12 @@ def _bind_iteration_rows( ``for entity, suffix in PAIRS`` with ``PAIRS = (("person", "support_channel"),)`` binds ``entity``/``suffix`` to their per-position choice sets — natural declarative loop code, in - scope (sol #583 round 6). The result reports whether nonempty - static rows were recognized and whether every target position was - fully propagated. Resolvable string columns remain exact; dynamic - columns and every name in a star payload bind opaque. + scope (sol #583 round 6). The result reports whether nonempty static + rows were recognized and whether the target geometry was completely + handled. Resolvable string columns remain exact and dynamic columns + bind opaque. A direct-name star is completely handled as opaque; + every name in a nested star payload is poisoned, but that geometry + remains partial and therefore enters the fragment fallback. """ literal = _static_iteration_value(iterable, self.constants) @@ -1639,6 +1650,8 @@ def _bind_iteration_target( target: ast.AST, values: tuple[str, ...] | None, ) -> bool: + """Bind name targets and report whether the geometry is all names.""" + for name in _assigned_names(target): self.bindings[-1][name] = None self.constants[-1][name] = ( @@ -3425,11 +3438,10 @@ def f(): def test_unpropagatable_geometry_over_guarded_fragments_fails_closed(): - """Sol #583 round-9: any target/container geometry the binder cannot - propagate fails closed AT THE LOOP when the container statically - carries guarded-name fragments — ending geometry-by-geometry chasing. - Dynamic iterables that merely mention strings (key lookups) and - fragment-free static tables stay clean.""" + """Sol #583 rounds 9-10: refused or partial binding over a supported + static container carrying guarded-name fragments fails closed at the + loop/comprehension generator. Dynamic expressions that merely mention + strings and fragment-free static tables stay clean.""" nested_star_payload = """ entity = "age" @@ -4023,7 +4035,7 @@ def f(df, columns): assert all("fail-closed" in item for item in dynamic_accesses) -def test_loop_targets_accept_every_static_string_iterable_form() -> None: +def test_loop_targets_accept_supported_static_string_iterable_forms() -> None: """Sets, dict keys, strings, and concatenation bind exact choices.""" benign_sources = ( From 35b28d3099761d3a657a8830262335d25a607d39 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 04:01:09 -0400 Subject: [PATCH 68/94] Add partial-binding precision regressions --- PROGRESS.md | 14 ++++-- .../tests/test_us_spine_blindness.py | 49 +++++++++++++++++++ 2 files changed, 60 insertions(+), 3 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 44175774..3fb47e9e 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -43,11 +43,19 @@ and partial structures propagated through bindings. supported scalar, structural-row, and static dict-view forms; helper docstrings distinguish exact propagation, deliberate opacity, and partial geometry; the round-9 claim now states the actual iteration-site fallback. +- Initial final validation passed (guard 121, focused graph 22, full + `populace-build` 3,343 passed/85 skipped, repository ruff), but the + independent net-diff audit found two precision defects before handoff: + partial Name/non-name targets retained stale outer constants, and opaque + constructor keys could collapse through the singleton sentinel. +- Added red controls for both audit findings plus an every-name nested-star + opacity check. The two precision controls fail on the current implementation + as expected; the nested-star check already passes. ## Next -- Run the guard file, full `populace-build` suite, repository ruff, and the - requested `acs_transfer` plus `congressional_district_vintage` cleanliness - checks. +- Propagate direct-name columns while poisoning unsupported partial targets, + and reject opaque dict-constructor entries before materializing a mapping. +- Re-run the full validation matrix after those corrections. - Restore `PROGRESS.md` exactly to `origin/main`, commit all coherent steps locally without pushing, and write `/private/tmp/583_fix5_handoff.md`. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index c5d002b9..af2579bf 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -3484,6 +3484,29 @@ def f(): assert any( "unpropagatable target geometry" in access for access in nested_star_accesses ) + nested_star_name_uses = """ +entity = "age" +middle = "income" +ROWS = (("person", "support", "channel"),) + + +def f(df): + return [ + ( + df.filter(items=[f"{entity}_support_channel"]), + df.filter(items=[f"person_{middle}_channel"]), + ) + for *(entity, middle), suffix in ROWS + ] +""" + nested_star_name_accesses = _source_spine_accesses(nested_star_name_uses) + assert ( + sum( + ".filter(items=...)" in access and "unresolvable" in access + for access in nested_star_name_accesses + ) + == 2 + ) for source in (rows_of_rows, dict_constructor): assert _source_spine_accesses(source), source assert _source_spine_accesses(dynamic_mentioning_strings) == () @@ -3514,6 +3537,20 @@ def test_non_name_targets_over_guarded_static_rows_fail_closed_at_loop(): source ) + fragment_free_partial_target = """ +entity = "person" + + +class Row: + pass + + +row = Row() +for entity, row.suffix in (("state", "fips"),): + sink(f"{entity}_support_channel") +""" + assert _source_spine_accesses(fragment_free_partial_target) == () + def test_static_dict_values_and_bound_partial_rows_propagate_per_column(): """Sol #583 round-10: static mapping values and partial structures @@ -3596,6 +3633,18 @@ def f(row): for source in key_only_fragment_controls: assert _source_spine_accesses(source) == (), source + dynamic_constructor_keys = """ +def f(key1, key2): + for entity, suffix in dict( + [ + (key1, ("person", "support_channel")), + (key2, ("state", "fips")), + ] + ).values(): + sink(f"{entity}_{suffix}") +""" + assert _source_spine_accesses(dynamic_constructor_keys) + def test_mid_star_rows_and_concatenated_dict_entries_are_in_scope(): """Sol #583 round-8 module-local edges: a star in any single target From e031abad1425ce06c209d9548ba76297dc9592ba Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 04:03:00 -0400 Subject: [PATCH 69/94] Preserve precision in partial static bindings --- PROGRESS.md | 12 ++- .../tests/test_us_spine_blindness.py | 100 +++++++++++++----- 2 files changed, 86 insertions(+), 26 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 3fb47e9e..51ccff66 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -51,11 +51,19 @@ and partial structures propagated through bindings. - Added red controls for both audit findings plus an every-name nested-star opacity check. The two precision controls fail on the current implementation as expected; the nested-star check already passes. +- Corrected partial-row handling so direct-name positions still receive their + exact column choices while unsupported subtargets are poisoned and keep the + overall binding partial. This removes stale outer constants without + weakening the fragment-bearing loop fallback, including star-only payloads. +- Narrowed `dict(iterable)` resolution to fully resolved entries; any opaque + key/value now refuses construction instead of collapsing distinct runtime + keys through the shared sentinel, leaving the syntactic value-only fragment + fallback to catch guarded content. +- The guard (121), focused graph battery (22), explicit `acs_transfer.py` and + `congressional_district_vintage.py` scans, and file-scoped ruff pass again. ## Next -- Propagate direct-name columns while poisoning unsupported partial targets, - and reject opaque dict-constructor entries before materializing a mapping. - Re-run the full validation matrix after those corrections. - Restore `PROGRESS.md` exactly to `origin/main`, commit all coherent steps locally without pushing, and write `/private/tmp/583_fix5_handoff.md`. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index af2579bf..0871f476 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -1182,17 +1182,30 @@ def _static_dict_value( ): return None entries = _static_literal_value(node.args[0], constants) - if entries is _OPAQUE_STATIC_VALUE: - structure = _static_structure(node.args[0], constants) - if structure is None: - return None - entries = structure + if _contains_opaque_static_value(entries): + return None try: return dict(entries) except (TypeError, ValueError): return None +def _contains_opaque_static_value(value: object) -> bool: + """Whether a partially resolved structure contains the opaque sentinel.""" + + if value is _OPAQUE_STATIC_VALUE: + return True + if isinstance(value, (list, tuple, set, frozenset)): + return any(_contains_opaque_static_value(item) for item in value) + if isinstance(value, dict): + return any( + _contains_opaque_static_value(item) + for pair in value.items() + for item in pair + ) + return False + + def _static_dict_values_expression(node: ast.AST) -> ast.AST | None: """Return only the syntactic value payload of a supported static dict.""" @@ -1562,6 +1575,15 @@ def visit_NamedExpr(self, node: ast.NamedExpr) -> None: ) self._bind([node.target], node.value, scope_index=scope_index) + def _poison_iteration_target_names(self, target: ast.AST) -> None: + """Bind every stored name below an unsupported target as opaque.""" + + for descendant in ast.walk(target): + if isinstance(descendant, ast.Name) and isinstance( + descendant.ctx, ast.Store + ): + self._bind_iteration_target(descendant, None) + def _bind_iteration_rows( self, target: ast.AST, @@ -1591,14 +1613,18 @@ def _bind_iteration_rows( if not rows or not all(isinstance(row, (list, tuple)) for row in rows): return False, False if not isinstance(target, (ast.Tuple, ast.List)): + self._poison_iteration_target_names(target) return True, False elements = list(target.elts) + if not elements: + return True, False star_positions = [ index for index, element in enumerate(elements) if isinstance(element, ast.Starred) ] if len(star_positions) > 1: + self._poison_iteration_target_names(target) return True, False if star_positions: # A star ANYWHERE absorbs mixed widths; names before it bind @@ -1611,39 +1637,42 @@ def _bind_iteration_rows( else: leading = elements trailing = [] - if not (leading or trailing) or not all( - isinstance(element, ast.Name) for element in (*leading, *trailing) - ): - return True, False needed = len(leading) + len(trailing) if star_positions: if any(len(row) < needed for row in rows): + self._poison_iteration_target_names(target) return True, False elif any(len(row) != len(leading) for row in rows): + self._poison_iteration_target_names(target) return True, False def _column_choices(values: tuple) -> tuple[str, ...] | None: return values if all(isinstance(value, str) for value in values) else None + fully_propagated = True for position, element in enumerate(leading): - self._bind_iteration_target( - element, - _column_choices(tuple(row[position] for row in rows)), - ) + if isinstance(element, ast.Name): + self._bind_iteration_target( + element, + _column_choices(tuple(row[position] for row in rows)), + ) + else: + self._poison_iteration_target_names(element) + fully_propagated = False for back, element in enumerate(reversed(trailing), start=1): - self._bind_iteration_target( - element, - _column_choices(tuple(row[-back] for row in rows)), - ) + if isinstance(element, ast.Name): + self._bind_iteration_target( + element, + _column_choices(tuple(row[-back] for row in rows)), + ) + else: + self._poison_iteration_target_names(element) + fully_propagated = False if not star_positions: - return True, True + return True, fully_propagated star_payload = elements[star_positions[0]].value - for descendant in ast.walk(star_payload): - if isinstance(descendant, ast.Name) and isinstance( - descendant.ctx, ast.Store - ): - self._bind_iteration_target(descendant, None) - return True, isinstance(star_payload, ast.Name) + self._poison_iteration_target_names(star_payload) + return True, fully_propagated and isinstance(star_payload, ast.Name) def _bind_iteration_target( self, @@ -3507,6 +3536,29 @@ def f(df): ) == 2 ) + star_only_nested_payload = """ +entity = "age" +middle = "income" +ROWS = (("person", "support"),) + + +def f(df): + return [ + ( + df.filter(items=[f"{entity}_support_channel"]), + df.filter(items=[f"person_{middle}_channel"]), + ) + for *(entity, middle), in ROWS + ] +""" + star_only_accesses = _source_spine_accesses(star_only_nested_payload) + assert ( + sum( + ".filter(items=...)" in access and "unresolvable" in access + for access in star_only_accesses + ) + == 2 + ) for source in (rows_of_rows, dict_constructor): assert _source_spine_accesses(source), source assert _source_spine_accesses(dynamic_mentioning_strings) == () From 56a68d8aa4b6325be1060fc6cf38da9ba6b2bd9c Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 04:09:08 -0400 Subject: [PATCH 70/94] Add static view composition regressions --- PROGRESS.md | 9 ++ .../tests/test_us_spine_blindness.py | 94 ++++++++++++++++++- 2 files changed, 102 insertions(+), 1 deletion(-) diff --git a/PROGRESS.md b/PROGRESS.md index 51ccff66..5209af00 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -61,9 +61,18 @@ and partial structures propagated through bindings. fallback to catch guarded content. - The guard (121), focused graph battery (22), explicit `acs_transfer.py` and `congressional_district_vintage.py` scans, and file-scoped ruff pass again. +- The last composition audit then identified three related cases not covered + by the exact reviewer inputs: scalar static `.values()` unnecessarily + entered the loop fallback, known strings disappeared from mixed columns, + and partial dict/constructor views were not retained through bindings. +- Added red precision/composition controls for literal, bound, and constructor + scalar views; inline/bound mixed columns; and bound partial dict views. ## Next +- Preserve partial dict entries without materializing opaque keys, feed scalar + dict views into exact string choices, and retain known strings in mixed + columns. - Re-run the full validation matrix after those corrections. - Restore `PROGRESS.md` exactly to `origin/main`, commit all coherent steps locally without pushing, and write `/private/tmp/583_fix5_handoff.md`. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 0871f476..f938cd4c 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -3685,6 +3685,40 @@ def f(row): for source in key_only_fragment_controls: assert _source_spine_accesses(source) == (), source + scalar_values_controls = ( + """ +def f(): + for entity in {"row": "person"}.values(): + sink(entity) +""", + """ +ROWS = {"row": "person"} + + +def f(): + for entity in ROWS.values(): + sink(entity) +""", + """ +def f(): + for entity in dict([("row", "person")]).values(): + sink(entity) +""", + ) + for source in scalar_values_controls: + assert _source_spine_accesses(source) == (), source + + scalar_values_guarded = """ +def f(): + for entity in {"row": "person"}.values(): + sink(f"{entity}_support_channel") +""" + scalar_accesses = _source_spine_accesses(scalar_values_guarded) + assert any("person_support_channel" in access for access in scalar_accesses) + assert not any( + "unpropagatable target geometry" in access for access in scalar_accesses + ) + dynamic_constructor_keys = """ def f(key1, key2): for entity, suffix in dict( @@ -3695,7 +3729,65 @@ def f(key1, key2): ).values(): sink(f"{entity}_{suffix}") """ - assert _source_spine_accesses(dynamic_constructor_keys) + dynamic_key_accesses = _source_spine_accesses(dynamic_constructor_keys) + assert any("person_support_channel" in access for access in dynamic_key_accesses) + assert not any( + "unpropagatable target geometry" in access for access in dynamic_key_accesses + ) + + mixed_column_rows = ( + """ +def f(dynamic_object): + for entity, suffix in ( + ("person", "support_channel"), + ("state", dynamic_object), + ): + sink(f"{entity}_{suffix}") +""", + """ +def f(dynamic_object): + rows = ( + ("person", "support_channel"), + ("state", dynamic_object), + ) + for entity, suffix in rows: + sink(f"{entity}_{suffix}") +""", + ) + for source in mixed_column_rows: + accesses = _source_spine_accesses(source) + assert any("person_support_channel" in access for access in accesses), source + assert not any( + "unpropagatable target geometry" in access for access in accesses + ), source + + bound_partial_dict_views = ( + """ +def f(dynamic_object): + rows = {"row": ("person", dynamic_object, "support_channel")} + for entity, obj, suffix in rows.values(): + sink(f"{entity}_{suffix}") +""", + """ +def f(dynamic_object): + entries = [("row", ("person", dynamic_object, "support_channel"))] + for entity, obj, suffix in dict(entries).values(): + sink(f"{entity}_{suffix}") +""", + """ +def f(dynamic_object): + entries = [("row", ("person", dynamic_object, "support_channel"))] + rows = dict(entries) + for entity, obj, suffix in rows.values(): + sink(f"{entity}_{suffix}") +""", + ) + for source in bound_partial_dict_views: + accesses = _source_spine_accesses(source) + assert any("person_support_channel" in access for access in accesses), source + assert not any( + "unpropagatable target geometry" in access for access in accesses + ), source def test_mid_star_rows_and_concatenated_dict_entries_are_in_scope(): From 0d0d39aae8b6824e917980b1f13fb7faedc74d9b Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 04:13:31 -0400 Subject: [PATCH 71/94] Compose partial structures through static dict views --- PROGRESS.md | 14 +- .../tests/test_us_spine_blindness.py | 140 +++++++++++++++--- 2 files changed, 133 insertions(+), 21 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 5209af00..f8f5d904 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -67,12 +67,20 @@ and partial structures propagated through bindings. and partial dict/constructor views were not retained through bindings. - Added red precision/composition controls for literal, bound, and constructor scalar views; inline/bound mixed columns; and bound partial dict views. +- Added an abstract ordered dict-entry representation: fully known keys retain + normal dict overwrite semantics, while opaque keys remain distinct possible + rows instead of collapsing through the sentinel. Partial literal and + constructor mappings now survive assignment bindings and feed their actual + keys, items, or values to iteration. +- Static iteration now retains known top-level string choices for scalar views + and known string members within mixed row columns. Empty views preserve + empty-loop flow, all-opaque columns remain opaque, and the documented + conservative duplicate-key over-catch applies only to unresolved keys. +- The expanded controls, 121-case guard, 22-case graph battery, and direct + production-file scans pass with both named files still at zero findings. ## Next -- Preserve partial dict entries without materializing opaque keys, feed scalar - dict views into exact string choices, and retain known strings in mixed - columns. - Re-run the full validation matrix after those corrections. - Restore `PROGRESS.md` exactly to `origin/main`, commit all coherent steps locally without pushing, and write `/private/tmp/583_fix5_handoff.md`. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index f938cd4c..b23abeac 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -36,7 +36,10 @@ Where multi-value loop bindings combine in one template, the guard may over-report combinations that no single row produces (a Cartesian over-catch). Over-reporting is the safe failure direction for a -tripwire; a module that trips it restructures its table. +tripwire; a module that trips it restructures its table. Partial dict +views likewise retain every entry with an opaque key because unknown +runtime keys may be distinct; if they collide, a value overwritten at +runtime can remain in the guard's conservative choice set. """ from __future__ import annotations @@ -515,6 +518,10 @@ class _StaticStringChoices(tuple): """Abstract alternatives bound by one static loop/comprehension target.""" +class _StaticDictEntries(tuple): + """Ordered abstract dict entries; opaque keys remain distinct rows.""" + + def _static_format_value( node: ast.AST, constants: list[dict[str, object]], @@ -1164,25 +1171,86 @@ def _static_structure( return None -def _static_dict_value( +def _static_dict_entries( node: ast.AST, constants: list[dict[str, object]], -) -> dict | None: - """Resolve a literal/bound dict or one supported ``dict(iterable)``.""" +) -> _StaticDictEntries | None: + """Resolve supported dict entries, conservatively preserving opaque keys.""" mapping = _static_literal_value(node, constants) - if isinstance(mapping, dict): + if isinstance(mapping, _StaticDictEntries): return mapping - if not ( + if isinstance(mapping, dict): + return _StaticDictEntries(mapping.items()) + if isinstance(node, ast.Dict): + if any(key is None for key in node.keys): + return None + entries = _StaticDictEntries( + ( + _static_value_or_structure(key, constants), + _static_value_or_structure(value, constants), + ) + for key, value in zip(node.keys, node.values, strict=True) + if key is not None + ) + elif ( isinstance(node, ast.Call) and isinstance(node.func, ast.Name) and node.func.id == "dict" and len(node.args) == 1 and not node.keywords ): + raw_entries = _static_value_or_structure(node.args[0], constants) + if isinstance(raw_entries, _StaticDictEntries): + entries = raw_entries + elif isinstance(raw_entries, dict): + entries = _StaticDictEntries(raw_entries.items()) + elif isinstance(raw_entries, (list, tuple, set, frozenset)): + pairs: list[tuple[object, object]] = [] + for entry in raw_entries: + if not isinstance(entry, (list, tuple)) or len(entry) != 2: + return None + pairs.append((entry[0], entry[1])) + entries = _StaticDictEntries(pairs) + else: + return None + else: + return None + + if any(_contains_opaque_static_value(key) for key, _value in entries): + # Unknown keys may or may not collide at runtime. Retain every row + # rather than materializing them through one shared sentinel. + return entries + try: + return _StaticDictEntries(dict(entries).items()) + except (TypeError, ValueError): return None - entries = _static_literal_value(node.args[0], constants) - if _contains_opaque_static_value(entries): + + +def _static_value_or_structure( + node: ast.AST, + constants: list[dict[str, object]], +) -> object: + """Resolve a full literal or retain its supported partial structure.""" + + value = _static_literal_value(node, constants) + if value is not _OPAQUE_STATIC_VALUE: + return value + structure = _static_structure(node, constants) + if structure is not None: + return structure + entries = _static_dict_entries(node, constants) + return _OPAQUE_STATIC_VALUE if entries is None else entries + + +def _static_dict_value( + node: ast.AST, + constants: list[dict[str, object]], +) -> dict | None: + """Resolve a fully static dict or supported ``dict(iterable)``.""" + + entries = _static_dict_entries(node, constants) + if entries is None or _contains_opaque_static_value(entries): return None try: return dict(entries) @@ -1279,15 +1347,33 @@ def _static_iteration_value( and not node.args and not node.keywords ): - mapping = _static_dict_value(node.func.value, constants) - if mapping is None: + entries = _static_dict_entries(node.func.value, constants) + if entries is None: return _OPAQUE_STATIC_VALUE return ( - tuple(mapping.items()) + tuple(entries) if node.func.attr == "items" - else tuple(mapping.values()) + else tuple(value for _key, value in entries) ) - return _static_literal_value(node, constants) + value = _static_literal_value(node, constants) + if isinstance(value, _StaticDictEntries): + return tuple(key for key, _value in value) + return value + + +def _static_iteration_string_choices( + node: ast.AST, + constants: list[dict[str, object]], +) -> tuple[str, ...] | None: + """Known string members of a supported static iteration value.""" + + value = _static_iteration_value(node, constants) + if not isinstance(value, (list, tuple, set, frozenset)): + return None + if not value: + return () + choices = tuple(item for item in value if isinstance(item, str)) + return choices or None def _iterable_carries_guarded_fragments(node: ast.AST) -> bool: @@ -1466,6 +1552,10 @@ def _bind_name( literal = _static_literal_value(value, self.constants) if literal is not _OPAQUE_STATIC_VALUE: constant = literal + if constant is None: + constant = _static_dict_value(value, self.constants) + if constant is None: + constant = _static_dict_entries(value, self.constants) if constant is None: constant = _static_structure(value, self.constants) if constant is None: @@ -1596,10 +1686,11 @@ def _bind_iteration_rows( per-position choice sets — natural declarative loop code, in scope (sol #583 round 6). The result reports whether nonempty static rows were recognized and whether the target geometry was completely - handled. Resolvable string columns remain exact and dynamic columns - bind opaque. A direct-name star is completely handled as opaque; - every name in a nested star payload is poisoned, but that geometry - remains partial and therefore enters the fragment fallback. + handled. Known string members of each column remain exact choices; a + column with no known strings binds opaque. A direct-name star is + completely handled as opaque; every name in a nested star payload is + poisoned, but that geometry remains partial and therefore enters the + fragment fallback. """ literal = _static_iteration_value(iterable, self.constants) @@ -1647,7 +1738,8 @@ def _bind_iteration_rows( return True, False def _column_choices(values: tuple) -> tuple[str, ...] | None: - return values if all(isinstance(value, str) for value in values) else None + choices = tuple(value for value in values if isinstance(value, str)) + return choices or None fully_propagated = True for position, element in enumerate(leading): @@ -1864,6 +1956,8 @@ def visit_For(self, node: ast.For) -> None: self._visit_access_target(node.target) before = self._flow_state() values = _static_string_list(node.iter, self.constants) + if values is None: + values = _static_iteration_string_choices(node.iter, self.constants) carries_guarded_fragments = _iterable_carries_guarded_fragments( node.iter ) or _resolved_value_carries_guarded_fragments(node.iter, self.constants) @@ -1909,6 +2003,11 @@ def _visit_comprehension( first_generator = node.generators[0] self.visit(first_generator.iter) first_values = _static_string_list(first_generator.iter, self.constants) + if first_values is None: + first_values = _static_iteration_string_choices( + first_generator.iter, + self.constants, + ) direct_class_body = bool( self.class_lexical_scope_depths and len(self.scope_kinds) == self.class_lexical_scope_depths[-1] @@ -1929,6 +2028,11 @@ def _visit_comprehension( if index: self.visit(generator.iter) values = _static_string_list(generator.iter, self.constants) + if values is None: + values = _static_iteration_string_choices( + generator.iter, + self.constants, + ) self._visit_access_target(generator.target) carries_guarded_fragments = _iterable_carries_guarded_fragments( generator.iter From bf10f339ed33e61d57cde93e9539142bddc0431c Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 04:16:52 -0400 Subject: [PATCH 72/94] Record round 10 validation receipts --- PROGRESS.md | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index f8f5d904..7b01313f 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,11 +2,12 @@ ## State -PR #583 round-10 fix lane is in progress on -`multispine-pool-build-578` from `9e9acf5`. The review HOLD identifies four -mechanical closure gaps in the spine-blindness structural guard: nested-star +PR #583 round-10 fixes are implemented and validated locally on +`multispine-pool-build-578` from `9e9acf5`. The review HOLD's four mechanical +closure gaps in the spine-blindness structural guard are closed: nested-star payload poisoning, non-name loop targets, static-dict `.values()` iteration, -and partial structures propagated through bindings. +and partial structures propagated through bindings. The branch has not been +pushed. ## Done @@ -78,9 +79,19 @@ and partial structures propagated through bindings. conservative duplicate-key over-catch applies only to unresolved keys. - The expanded controls, 121-case guard, 22-case graph battery, and direct production-file scans pass with both named files still at zero findings. +- Re-ran the complete validation matrix at `0d0d39a`: the guard passes 121, + the focused graph battery passes 22, and the full `populace-build` suite + passes 3,343 with 85 skipped and five existing warnings. Repository-wide + ruff, file formatting, and `git diff --check` pass. +- Authenticated the six reviewer repros directly at the same HEAD: the nested + star, attribute target, and subscript target fail closed at iteration; the + literal `.values()`, constructor `.values()`, and bound mixed-row forms + resolve the exact `person_support_channel` column. Direct scans of + `acs_transfer.py` and `congressional_district_vintage.py` remain empty. +- Confirmed `origin/multispine-pool-build-578` still points to the requested + starting commit `9e9acf5`; all work remains local. ## Next -- Re-run the full validation matrix after those corrections. -- Restore `PROGRESS.md` exactly to `origin/main`, commit all coherent steps +- Restore `PROGRESS.md` exactly to `origin/main`, commit the restoration locally without pushing, and write `/private/tmp/583_fix5_handoff.md`. From 93b0b8dfe236c1147b2efb7d130c8567dc182eef Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 04:17:20 -0400 Subject: [PATCH 73/94] Restore root progress journal --- PROGRESS.md | 125 ++++++++++++++++------------------------------------ 1 file changed, 38 insertions(+), 87 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 7b01313f..f3158fdd 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,96 +2,47 @@ ## State -PR #583 round-10 fixes are implemented and validated locally on -`multispine-pool-build-578` from `9e9acf5`. The review HOLD's four mechanical -closure gaps in the spine-blindness structural guard are closed: nested-star -payload poisoning, non-name loop targets, static-dict `.values()` iteration, -and partial structures propagated through bindings. The branch has not been -pushed. +Populace #516 whole-row donor outlier screen is complete on +`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515 +interim carve merged as #525). The `puf_tax_detail` donor now drops tax units +whose grouped raw mortgage interest reaches $10M before the #515 carve +(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T +of phantom mortgage-interest mass), with the checkpoint schema bumped to v3 +so post-carve pre-screen checkpoints rebuild. ## Done -- Confirmed the requested branch, clean worktree, and exact starting commit. -- Read `CLAUDE.md`, the GitNexus debugging skill, and the authenticated - round-10 review log. -- Recorded the required implementation, regression, documentation, precision, - suite, lint, graph-cleanliness, and no-push constraints. -- Confirmed the GitNexus CLI is installed but the repository has no index; a - non-augmenting index attempt was blocked by the managed global-registry - write, and its generated untracked local index was removed. -- Reproduced all six authenticated forms as silent at the starting guard: - stale-outer nested star, attribute target, subscript target, literal - `.values()`, constructor `.values()`, and a bound mixed row. -- Added those exact inputs as self-tests, with loop-diagnostic assertions for - unpropagatable geometry and exact-column assertions for successful - per-column propagation. The focused red run fails all three test groups as - expected before the implementation. -- Reworked static-row binding to report recognized and fully propagated - status separately. Nested star payload names are all poisoned, partial and - refused targets now enter the loop fallback regardless of flattened-string - resolution, and fragment state is captured before target bindings mutate - the scope. -- Added static `.items()`/`.values()` iteration resolution for literal, - bound, and supported `dict(...)` mappings, with value-only syntactic - fragment probing so guarded-looking mapping keys do not taint benign - `.values()` loops. -- Preserved partial list/tuple structures at assignment bind time, allowing - bound mixed rows to propagate their static columns exactly. -- The 121-case guard and focused 22-case benign/runtime/graph battery pass. - Direct scans report zero findings for both `acs_transfer.py` and - `congressional_district_vintage.py`; file-scoped ruff and formatting pass. -- Completed the docstring truth pass: the module contract now names the - supported scalar, structural-row, and static dict-view forms; helper - docstrings distinguish exact propagation, deliberate opacity, and partial - geometry; the round-9 claim now states the actual iteration-site fallback. -- Initial final validation passed (guard 121, focused graph 22, full - `populace-build` 3,343 passed/85 skipped, repository ruff), but the - independent net-diff audit found two precision defects before handoff: - partial Name/non-name targets retained stale outer constants, and opaque - constructor keys could collapse through the singleton sentinel. -- Added red controls for both audit findings plus an every-name nested-star - opacity check. The two precision controls fail on the current implementation - as expected; the nested-star check already passes. -- Corrected partial-row handling so direct-name positions still receive their - exact column choices while unsupported subtargets are poisoned and keep the - overall binding partial. This removes stale outer constants without - weakening the fragment-bearing loop fallback, including star-only payloads. -- Narrowed `dict(iterable)` resolution to fully resolved entries; any opaque - key/value now refuses construction instead of collapsing distinct runtime - keys through the shared sentinel, leaving the syntactic value-only fragment - fallback to catch guarded content. -- The guard (121), focused graph battery (22), explicit `acs_transfer.py` and - `congressional_district_vintage.py` scans, and file-scoped ruff pass again. -- The last composition audit then identified three related cases not covered - by the exact reviewer inputs: scalar static `.values()` unnecessarily - entered the loop fallback, known strings disappeared from mixed columns, - and partial dict/constructor views were not retained through bindings. -- Added red precision/composition controls for literal, bound, and constructor - scalar views; inline/bound mixed columns; and bound partial dict views. -- Added an abstract ordered dict-entry representation: fully known keys retain - normal dict overwrite semantics, while opaque keys remain distinct possible - rows instead of collapsing through the sentinel. Partial literal and - constructor mappings now survive assignment bindings and feed their actual - keys, items, or values to iteration. -- Static iteration now retains known top-level string choices for scalar views - and known string members within mixed row columns. Empty views preserve - empty-loop flow, all-opaque columns remain opaque, and the documented - conservative duplicate-key over-catch applies only to unresolved keys. -- The expanded controls, 121-case guard, 22-case graph battery, and direct - production-file scans pass with both named files still at zero findings. -- Re-ran the complete validation matrix at `0d0d39a`: the guard passes 121, - the focused graph battery passes 22, and the full `populace-build` suite - passes 3,343 with 85 skipped and five existing warnings. Repository-wide - ruff, file formatting, and `git diff --check` pass. -- Authenticated the six reviewer repros directly at the same HEAD: the nested - star, attribute target, and subscript target fail closed at iteration; the - literal `.values()`, constructor `.values()`, and bound mixed-row forms - resolve the exact `person_support_channel` column. Direct scans of - `acs_transfer.py` and `congressional_district_vintage.py` remain empty. -- Confirmed `origin/multispine-pool-build-578` still points to the requested - starting commit `9e9acf5`; all work remains local. +- Confirmed a clean starting worktree at `aef1c56`. +- Read the repository guidance and established the #515 donor carve as the + screen's required downstream boundary. +- Started source-level audits of every donor-frame consumer, checkpoint + validation, row-count pins, and existing donor-fact summaries. +- Attempted the requested GitNexus impact workflow; the managed filesystem + denied its global registry write. Its local index also exposed a broad + `build/` ignore mismatch, so the completed impact audit uses direct source + call sites and tests. +- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the + structural rationale and pinned-artifact receipts. +- Added a whole-row screen on grouped raw person `home_mortgage_interest` + after tax-unit assembly, before the #515 carve, with retained-index reset. +- Confirmed no downstream consumer pairs donor rows to the original HDF arrays + or carries a stale donor-length vector; values and weights always originate + from the same screened frame. +- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale + checkpoint regression track the live constant while retaining literal-v1 + corruptions. +- Added regression coverage for the exact grouped boundary, whole-row removal, + retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant. +- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets + 139; populace-data 138 with 1 skip. The directly affected tail-bound suite + adds 12 passes. Ruff format/check and `git diff --check` are clean. +- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line + audit, expected 208,611-row real-artifact effect, verification results, count + sweep, and deliberately untouched surfaces. ## Next -- Restore `PROGRESS.md` exactly to `origin/main`, commit the restoration - locally without pushing, and write `/private/tmp/583_fix5_handoff.md`. +- PR #527 review cycle, then merge. After both #525 and #527: rebuild the + base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a + run that holds per `us_critical_targets.py`. +- Root record-level ETL carve stays open on populace#515. From b32f20ed958c08d0b803b14191a2cdaf4eee72d8 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 04:49:28 -0400 Subject: [PATCH 74/94] Round 11: opaque-over-string positions are incomplete; static stars splice MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sol round-11 closure implemented at the root: (1) static starred expansions splice into rows during resolution, so value-side stars propagate precisely and catch by name; (2) propagation completeness now means every string-bearing position bound to real choices — a star swallowing strings at any depth (leaf-walked), or a string-bearing column with opaque members, marks the binding incomplete and fires the loop fail-closed over fragment-bearing containers, with dual reporting beside any named catch as the documented conservative direction. All-dynamic columns and dynamic-only star payloads stay complete, so the congressional-district vintage pattern remains clean. Two fix-5-era expectations reconciled to the closure; the round-7 benign-star control now uses fragment-free data. Co-Authored-By: Claude Fable 5 --- .../tests/test_us_spine_blindness.py | 66 ++++++++++++++----- 1 file changed, 49 insertions(+), 17 deletions(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index b23abeac..10c046e0 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -1161,7 +1161,13 @@ def _static_structure( if isinstance(node, (ast.List, ast.Tuple)): resolved = [] for element in node.elts: - if isinstance(element, (ast.List, ast.Tuple)): + if isinstance(element, ast.Starred): + inner_value = _static_literal_value(element.value, constants) + if isinstance(inner_value, (list, tuple)): + resolved.extend(inner_value) + else: + resolved.append(_OPAQUE_STATIC_VALUE) + elif isinstance(element, (ast.List, ast.Tuple)): inner = _static_structure(element, constants) resolved.append(inner if inner is not None else _OPAQUE_STATIC_VALUE) else: @@ -1737,26 +1743,27 @@ def _bind_iteration_rows( self._poison_iteration_target_names(target) return True, False - def _column_choices(values: tuple) -> tuple[str, ...] | None: + fully_propagated = True + + def _bind_column(element: ast.Name, values: tuple) -> None: + nonlocal fully_propagated choices = tuple(value for value in values if isinstance(value, str)) - return choices or None + self._bind_iteration_target(element, choices or None) + if choices and len(choices) != len(values): + # A string-bearing column with opaque members is + # incomplete propagation (sol #583 round 11); all-dynamic + # columns (no strings) stay complete-enough. + fully_propagated = False - fully_propagated = True for position, element in enumerate(leading): if isinstance(element, ast.Name): - self._bind_iteration_target( - element, - _column_choices(tuple(row[position] for row in rows)), - ) + _bind_column(element, tuple(row[position] for row in rows)) else: self._poison_iteration_target_names(element) fully_propagated = False for back, element in enumerate(reversed(trailing), start=1): if isinstance(element, ast.Name): - self._bind_iteration_target( - element, - _column_choices(tuple(row[-back] for row in rows)), - ) + _bind_column(element, tuple(row[-back] for row in rows)) else: self._poison_iteration_target_names(element) fully_propagated = False @@ -1764,7 +1771,27 @@ def _column_choices(values: tuple) -> tuple[str, ...] | None: return True, fully_propagated star_payload = elements[star_positions[0]].value self._poison_iteration_target_names(star_payload) - return True, fully_propagated and isinstance(star_payload, ast.Name) + + def _bears_strings(value: object) -> bool: + if isinstance(value, str): + return True + if isinstance(value, (list, tuple, set, frozenset)): + return any(_bears_strings(item) for item in value) + if isinstance(value, dict): + return any( + _bears_strings(part) for part in (*value.keys(), *value.values()) + ) + return False + + swallowed_strings = any( + _bears_strings(value) + for row in rows + for value in row[len(leading) : len(row) - len(trailing)] + ) + # A star swallowing string positions hides column material behind + # an opaque name — incomplete propagation whatever the payload + # shape (sol #583 round 11). + return True, fully_propagated and not swallowed_strings def _bind_iteration_target( self, @@ -3557,8 +3584,12 @@ def f(): for entity, suffix, *rest in ROWS: sink(f"{entity}_{suffix}") """ + # Fragment-free rows: with guarded-name fragments present, a star + # swallowing any string now counts as incomplete propagation and + # fail-closes conservatively (round 11) — so the benign control uses + # data that carries no fragments at all. benign_starred = """ -ROWS = (("person", "age"), ("household", "weight", "x")) +ROWS = (("state", "age"), ("county", "weight", "x")) def f(): @@ -3861,9 +3892,10 @@ def f(dynamic_object): for source in mixed_column_rows: accesses = _source_spine_accesses(source) assert any("person_support_channel" in access for access in accesses), source - assert not any( - "unpropagatable target geometry" in access for access in accesses - ), source + # Round 11: a string-bearing column with opaque members counts as + # incomplete propagation, so the loop may ALSO record fail-closed + # beside the named catch — dual reporting is the conservative + # direction, not a defect. bound_partial_dict_views = ( """ From 6d903a6418eb80b3e47fc8ff1bf9d0590c9bf345 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 05:12:45 -0400 Subject: [PATCH 75/94] =?UTF-8?q?Round=2012:=20string=20material=20is=20al?= =?UTF-8?q?ways=20visible=20=E2=80=94=20structures=20propagate,=20never=20?= =?UTF-8?q?hide?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sol round-12 closure, resolved by propagating harder instead of fail-closing wider: (1) a column of one repeated static structure binds AS that structure and a new destructuring bind resolves a, b = payload per position, so nested rows catch by NAME; (2) columns of divergent structures bind their flattened string leaves as choices — every string stays visible in the binding, keeping entity->columns registry tables (acs_transfer, capital_gain_details, housing_inputs) complete and clean; (3) {**BASE} dict expansion merges statically; (4) the dual-report reconciliation now REQUIRES the fail-closed record beside the named catch; (5) helper and module docstrings state the completeness semantics exactly; (6) rounds 11-12 fixtures committed for provenance, including the value-side star splice catching by name. Co-Authored-By: Claude Fable 5 --- .../tests/test_us_spine_blindness.py | 195 +++++++++++++++++- 1 file changed, 184 insertions(+), 11 deletions(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 10c046e0..c9eb6153 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -2,7 +2,7 @@ This guard enforces, and its tests certify, exactly these surfaces: -- direct attribute, subscript, ``.loc``, and ``__getitem__`` reads; +- direct attribute, subscript, ``.loc``, and ``__getitem__`` reads and store targets; - canonical guarded-column factory calls, including aliases bound by simple assignment or named expression; - ``query``, ``eval``, ``filter``, and ``get`` expression surfaces, failing @@ -566,7 +566,26 @@ def _static_literal_value( return values if isinstance(node, ast.Dict): if any(key is None for key in node.keys): - return _OPAQUE_STATIC_VALUE + # {**BASE} expansion: merge statically resolvable mappings in + # order, like the runtime does (sol #583 round 12); any + # unresolvable expansion makes the whole dict opaque. + merged: dict = {} + for key, value in zip(node.keys, node.values, strict=True): + if key is None: + inner = _static_literal_value(value, constants) + if not isinstance(inner, dict): + return _OPAQUE_STATIC_VALUE + merged.update(inner) + else: + resolved_key = _static_literal_value(key, constants) + resolved_value = _static_literal_value(value, constants) + if _OPAQUE_STATIC_VALUE in (resolved_key, resolved_value): + return _OPAQUE_STATIC_VALUE + try: + merged[resolved_key] = resolved_value + except TypeError: + return _OPAQUE_STATIC_VALUE + return merged keys = tuple( _static_literal_value(key, constants) for key in node.keys @@ -1307,6 +1326,20 @@ def _static_dict_values_expression(node: ast.AST) -> ast.AST | None: return None +def _value_bears_strings(value: object) -> bool: + """True when a resolved value carries string material at any depth.""" + + if isinstance(value, str): + return True + if isinstance(value, (list, tuple, set, frozenset)): + return any(_value_bears_strings(item) for item in value) + if isinstance(value, dict): + return any( + _value_bears_strings(part) for part in (*value.keys(), *value.values()) + ) + return False + + def _static_container_expression(node: ast.AST) -> ast.AST | None: """The literal-container core of an iterable expression, if any. @@ -1640,8 +1673,33 @@ def visit_Assign(self, node: ast.Assign) -> None: self._visit_access_target(target) if isinstance(target, ast.Subscript) and isinstance(target.value, ast.Name): self._poison(target.value.id) + if self._bind_destructured(node.targets, node.value): + return self._bind(list(node.targets), node.value) + def _bind_destructured(self, targets: list[ast.AST], value: ast.AST) -> bool: + """Destructure ``a, b = payload`` when payload resolves to a static + string sequence — each name binds its exact position, so nested + structures propagated from row bindings resolve precisely + (sol #583 round 12).""" + + if len(targets) != 1 or not isinstance(targets[0], (ast.Tuple, ast.List)): + return False + elements = targets[0].elts + if not all(isinstance(element, ast.Name) for element in elements): + return False + resolved = _static_literal_value(value, self.constants) + if ( + resolved is _OPAQUE_STATIC_VALUE + or not isinstance(resolved, (list, tuple)) + or len(resolved) != len(elements) + or not all(isinstance(item, str) for item in resolved) + ): + return False + for element, item in zip(elements, resolved, strict=True): + self._bind_iteration_target(element, (item,)) + return True + def visit_AugAssign(self, node: ast.AugAssign) -> None: self.visit(node.value) self.visit(node.target) @@ -1691,11 +1749,14 @@ def _bind_iteration_rows( "support_channel"),)`` binds ``entity``/``suffix`` to their per-position choice sets — natural declarative loop code, in scope (sol #583 round 6). The result reports whether nonempty static - rows were recognized and whether the target geometry was completely - handled. Known string members of each column remain exact choices; a - column with no known strings binds opaque. A direct-name star is - completely handled as opaque; every name in a nested star payload is - poisoned, but that geometry remains partial and therefore enters the + rows were recognized and whether propagation was COMPLETE: every + string-bearing position bound to real choices. Known string members + of each column remain exact choices; a column whose values carry no + strings at any depth binds opaque and stays complete; a column with + nested or mixed string material, or a star swallowing string-bearing + values at any depth, is incomplete regardless of payload shape — + every name in any star payload is poisoned, and incomplete + propagation enters the fragment fallback. """ @@ -1747,9 +1808,52 @@ def _bind_iteration_rows( def _bind_column(element: ast.Name, values: tuple) -> None: nonlocal fully_propagated + structured = [ + value + for value in values + if not isinstance(value, str) and _value_bears_strings(value) + ] + if structured and len(structured) == len(values): + distinct = {repr(value) for value in structured} + if len(distinct) == 1: + # A column of one repeated static structure propagates + # AS that structure — downstream unpacking destructures + # it precisely (sol #583 round 12). + self._bind_iteration_target(element, None) + self.constants[-1][element.id] = structured[0] + return + + def _leaves(value: object) -> tuple[str, ...]: + if isinstance(value, str): + return (value,) + if isinstance(value, (list, tuple, set, frozenset)): + return tuple(leaf for item in value for leaf in _leaves(item)) + if isinstance(value, dict): + return tuple( + leaf + for part in (*value.keys(), *value.values()) + for leaf in _leaves(part) + ) + return () + + # Divergent structures bind their flattened string leaves + # as choices: every string stays VISIBLE in the binding, so + # nothing is hidden behind opacity and registry tables + # (entity -> column tuples) remain complete + # (sol #583 round 12). + self._bind_iteration_target( + element, + tuple(leaf for value in structured for leaf in _leaves(value)) + or None, + ) + return choices = tuple(value for value in values if isinstance(value, str)) self._bind_iteration_target(element, choices or None) - if choices and len(choices) != len(values): + if structured: + # Mixed flat/nested string material — string content + # reaches an opaque binding (sol #583 round 12). + fully_propagated = False + elif choices and len(choices) != len(values): # A string-bearing column with opaque members is # incomplete propagation (sol #583 round 11); all-dynamic # columns (no strings) stay complete-enough. @@ -3557,6 +3661,72 @@ def f(df): assert not _source_spine_accesses(benign_bound) +def test_round_11_and_12_string_material_never_reaches_opaque_bindings(): + """Sol #583 rounds 11-12 closure fixtures, committed for provenance: + stars swallowing strings, value-side starred splices, nested + string-bearing columns, and {**BASE} dict expansion all classify — + caught by name or loop-fail-closed — never silent.""" + + star_swallows_strings = """ +import pandas as pd + + +def f(df: pd.DataFrame): + for entity, *parts in [("person", "support", "channel")]: + return df[entity + "_" + "_".join(parts)] +""" + value_side_star = """ +import pandas as pd + + +def f(df: pd.DataFrame): + for entity, suffix in [("person", *("support_channel",))]: + return df[f"{entity}_{suffix}"] +""" + nested_column = """ +ROWS = ((("person", "support_channel"), "meta"),) + + +def f(): + for payload, meta in ROWS: + entity, suffix = payload + sink(f"{entity}_{suffix}") +""" + dict_expansion = """ +BASE = {"person": "support_channel"} +ROWS = {**BASE} + + +def f(): + for entity, suffix in ROWS.items(): + sink(f"{entity}_{suffix}") +""" + for source in ( + star_swallows_strings, + value_side_star, + nested_column, + dict_expansion, + ): + assert _source_spine_accesses(source), source + # The splice and the expansion resolve precisely — by name, not + # merely by opacity. + assert any( + "person_support_channel" in access + for access in _source_spine_accesses(value_side_star) + ) + assert any( + "person_support_channel" in access + for access in _source_spine_accesses(dict_expansion) + ) + # Dynamic-only star payloads and all-dynamic columns stay complete. + dynamic_star = """ +def f(a, b): + for label, *objs in (("state", a, b),): + sink(label) +""" + assert _source_spine_accesses(dynamic_star) == () + + def test_dict_items_and_starred_row_iteration_are_in_scope(): """Sol #583 round-7 module-local edges: static dict.items() and starred/mixed-width row unpacking are ordinary declarative code.""" @@ -3893,9 +4063,12 @@ def f(dynamic_object): accesses = _source_spine_accesses(source) assert any("person_support_channel" in access for access in accesses), source # Round 11: a string-bearing column with opaque members counts as - # incomplete propagation, so the loop may ALSO record fail-closed - # beside the named catch — dual reporting is the conservative - # direction, not a defect. + # incomplete propagation, so the loop MUST also record fail-closed + # beside the named catch — dual reporting is the required + # conservative direction (sol round 12). + assert any("unpropagatable target geometry" in access for access in accesses), ( + source + ) bound_partial_dict_views = ( """ From 1d439813a76b63928aecf19c58661928430ea5c7 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 05:25:28 -0400 Subject: [PATCH 76/94] Start PR 583 round 13 fix journal --- PROGRESS.md | 58 ++++++++++++++++++----------------------------------- 1 file changed, 20 insertions(+), 38 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index f3158fdd..d0ed2d76 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,47 +2,29 @@ ## State -Populace #516 whole-row donor outlier screen is complete on -`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515 -interim carve merged as #525). The `puf_tax_detail` donor now drops tax units -whose grouped raw mortgage interest reaches $10M before the #515 carve -(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T -of phantom mortgage-interest mass), with the checkpoint schema bumped to v3 -so post-carve pre-screen checkpoints rebuild. +PR #583 round-13 fix lane is in progress on +`multispine-pool-build-578` from `6d903a6`. The review HOLD identifies one +resolver divergence: the loop fail-closed trigger and fragment/value probes do +not use the binder's structure-based resolution, so starred wrappers and +partially static dict merges can hide guarded fragments or suppress required +loop records. ## Done -- Confirmed a clean starting worktree at `aef1c56`. -- Read the repository guidance and established the #515 donor carve as the - screen's required downstream boundary. -- Started source-level audits of every donor-frame consumer, checkpoint - validation, row-count pins, and existing donor-fact summaries. -- Attempted the requested GitNexus impact workflow; the managed filesystem - denied its global registry write. Its local index also exposed a broad - `build/` ignore mismatch, so the completed impact audit uses direct source - call sites and tests. -- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the - structural rationale and pinned-artifact receipts. -- Added a whole-row screen on grouped raw person `home_mortgage_interest` - after tax-unit assembly, before the #515 carve, with retained-index reset. -- Confirmed no downstream consumer pairs donor rows to the original HDF arrays - or carries a stale donor-length vector; values and weights always originate - from the same screened frame. -- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale - checkpoint regression track the live constant while retaining literal-v1 - corruptions. -- Added regression coverage for the exact grouped boundary, whole-row removal, - retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant. -- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets - 139; populace-data 138 with 1 skip. The directly affected tail-bound suite - adds 12 passes. Ruff format/check and `git diff --check` are clean. -- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line - audit, expected 208,611-row real-artifact effect, verification results, count - sweep, and deliberately untouched surfaces. +- Confirmed the requested branch, clean worktree, and exact starting commit. +- Read `CLAUDE.md`, the GitNexus debugging skill, and the round-13 review HOLD + at `/Users/maxghenis/PolicyEngine/_buildo-runtime/reviews/sol_583_r13.log`. +- Recorded the required shared structure resolution, partial-dict retention, + dual-report, fixture-backfill, precision, suite, lint, no-push, root-journal, + and handoff constraints. ## Next -- PR #527 review cycle, then merge. After both #525 and #527: rebuild the - base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a - run that holds per `us_critical_targets.py`. -- Root record-level ETL carve stays open on populace#515. +- Trace the binder and all three fragment/loop probes, then commit the missing + round-12 mirror fixtures and all round-13 repro/comprehension mirrors. +- Route every probe through shared structure-based resolution, retaining + resolvable dict-merge entries alongside opaque sentinels. +- Run the guard file, precision and benign batteries, full `populace-build` + suite, repository Ruff, and `git diff --check`. +- Restore `PROGRESS.md` exactly to `origin/main`, commit every coherent step + locally without pushing, and write `/private/tmp/583_fix6_handoff.md`. From 1c4cb629d9387c04d7ebdd6a2861f53f5476576c Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 05:30:38 -0400 Subject: [PATCH 77/94] Backfill round 12 nested structure mirrors --- PROGRESS.md | 6 +- .../tests/test_us_spine_blindness.py | 59 +++++++++++++++++++ 2 files changed, 63 insertions(+), 2 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index d0ed2d76..25925992 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -17,11 +17,13 @@ loop records. - Recorded the required shared structure resolution, partial-dict retention, dual-report, fixture-backfill, precision, suite, lint, no-push, root-journal, and handoff constraints. +- Recovered the exact nine previously silent round-12 nested-structure mirrors + from the authenticated reviewer transcript and committed them as explicit + fail-closed regression fixtures. ## Next -- Trace the binder and all three fragment/loop probes, then commit the missing - round-12 mirror fixtures and all round-13 repro/comprehension mirrors. +- Commit all three round-13 repros and their comprehension mirrors. - Route every probe through shared structure-based resolution, retaining resolvable dict-merge entries alongside opaque sentinels. - Run the guard file, precision and benign batteries, full `populace-build` diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index c9eb6153..931b6cc9 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -3727,6 +3727,65 @@ def f(a, b): assert _source_spine_accesses(dynamic_star) == () +def test_round_12_nested_string_material_mirrors_fail_closed(): + """The exact nine round-12 review mirrors remain committed fixtures.""" + + sources = { + "literal_loop": """ +def f(dynamic): + for payload, suffix in ((("person", dynamic), "support_channel"),): + sink(f"{payload}_{suffix}") +""", + "bound_loop": """ +def f(dynamic): + rows = ((("person", dynamic), "support_channel"),) + for payload, suffix in rows: + sink(f"{payload}_{suffix}") +""", + "literal_comp": """ +def f(dynamic): + return [f"{payload}_{suffix}" for payload, suffix in ((("person", dynamic), "support_channel"),)] +""", + "bound_comp": """ +def f(dynamic): + rows = ((("person", dynamic), "support_channel"),) + return [f"{payload}_{suffix}" for payload, suffix in rows] +""", + "literal_values": """ +def f(dynamic): + for payload, suffix in {"row": (("person", dynamic), "support_channel")}.values(): + sink(f"{payload}_{suffix}") +""", + "bound_values": """ +def f(dynamic): + rows = {"row": (("person", dynamic), "support_channel")} + for payload, suffix in rows.values(): + sink(f"{payload}_{suffix}") +""", + "ctor_values": """ +def f(dynamic): + rows = dict([("row", (("person", dynamic), "support_channel"))]) + for payload, suffix in rows.values(): + sink(f"{payload}_{suffix}") +""", + "literal_items": """ +def f(dynamic): + for payload, suffix in {("person", dynamic): "support_channel"}.items(): + sink(f"{payload}_{suffix}") +""", + "bound_items_comp": """ +def f(dynamic): + rows = {("person", dynamic): "support_channel"} + return [f"{payload}_{suffix}" for payload, suffix in rows.items()] +""", + } + for name, source in sources.items(): + accesses = _source_spine_accesses(source) + assert any( + "unpropagatable target geometry" in access for access in accesses + ), name + + def test_dict_items_and_starred_row_iteration_are_in_scope(): """Sol #583 round-7 module-local edges: static dict.items() and starred/mixed-width row unpacking are ordinary declarative code.""" From 1e89916889df456aea527b00cabb731767aaf164 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 05:31:08 -0400 Subject: [PATCH 78/94] Add round 13 structure resolver regressions --- PROGRESS.md | 4 +- .../tests/test_us_spine_blindness.py | 86 +++++++++++++++++++ 2 files changed, 89 insertions(+), 1 deletion(-) diff --git a/PROGRESS.md b/PROGRESS.md index 25925992..d2c60795 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -20,10 +20,12 @@ loop records. - Recovered the exact nine previously silent round-12 nested-structure mirrors from the authenticated reviewer transcript and committed them as explicit fail-closed regression fixtures. +- Added the three exact round-13 loop repros and their comprehension mirrors, + pinning loop-only classification for the starred target and dual reports for + the partial dict merge and dynamic-tail starred rows. ## Next -- Commit all three round-13 repros and their comprehension mirrors. - Route every probe through shared structure-based resolution, retaining resolvable dict-merge entries alongside opaque sentinels. - Run the guard file, precision and benign batteries, full `populace-build` diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 931b6cc9..dd763bd5 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -3786,6 +3786,92 @@ def f(dynamic): ), name +def test_starred_iterable_wrappers_fail_closed_for_star_targets(): + """Sol #583 round 13: iterable stars use the binder's fragment view.""" + + loop = """ +ROWS = (("person", "support", "channel"),) + + +def f(): + for entity, *parts in [*ROWS]: + sink(entity + "_" + "_".join(parts)) +""" + comprehension = """ +ROWS = (("person", "support", "channel"),) + + +def f(): + return [ + entity + "_" + "_".join(parts) + for entity, *parts in [*ROWS] + ] +""" + for source in (loop, comprehension): + accesses = _source_spine_accesses(source) + assert any( + "unpropagatable target geometry" in access for access in accesses + ), source + + +def test_partial_static_dict_merges_retain_entries_and_incompleteness(): + """Sol #583 round 13: known merge entries survive dynamic siblings.""" + + loop = """ +BASE = {"person": "support_channel"} + + +def f(dynamic): + for entity, suffix in {**BASE, "state": dynamic}.items(): + sink(f"{entity}_{suffix}") +""" + comprehension = """ +BASE = {"person": "support_channel"} + + +def f(dynamic): + return [ + f"{entity}_{suffix}" + for entity, suffix in {**BASE, "state": dynamic}.items() + ] +""" + for source in (loop, comprehension): + accesses = _source_spine_accesses(source) + assert any("person_support_channel" in access for access in accesses), source + assert any( + "unpropagatable target geometry" in access for access in accesses + ), source + + +def test_starred_static_rows_with_dynamic_tail_dual_report(): + """Sol #583 round 13: starred rows retain the required dual report.""" + + loop = """ +ROWS = (("person", "support_channel"),) + + +def f(dynamic): + for entity, suffix in [*ROWS, ("state", dynamic)]: + sink(f"{entity}_{suffix}") +""" + comprehension = """ +ROWS = (("person", "support_channel"),) + + +def f(dynamic): + return [ + f"{entity}_{suffix}" + for entity, suffix in [*ROWS, ("state", dynamic)] + ] +""" + for source in (loop, comprehension): + accesses = _source_spine_accesses(source) + assert any("person_support_channel" in access for access in accesses), source + assert any( + "unpropagatable target geometry" in access for access in accesses + ), source + + def test_dict_items_and_starred_row_iteration_are_in_scope(): """Sol #583 round-7 module-local edges: static dict.items() and starred/mixed-width row unpacking are ordinary declarative code.""" From 942030fec346deccd003f34a6941b4952e843e52 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 05:34:32 -0400 Subject: [PATCH 79/94] Unify static iteration structure resolution --- PROGRESS.md | 8 +- .../tests/test_us_spine_blindness.py | 193 +++++++----------- 2 files changed, 77 insertions(+), 124 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index d2c60795..fe712441 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -23,11 +23,15 @@ loop records. - Added the three exact round-13 loop repros and their comprehension mirrors, pinning loop-only classification for the starred target and dual reports for the partial dict merge and dynamic-tail starred rows. +- Unified the binder and both fragment probes on one structure-based iterable + resolver, including starred splicing and dict-entry views. +- Preserved resolvable partial-dict merge entries beside opaque sentinels and + retained partial nested structures while marking their bindings incomplete. +- Confirmed all 15 new loop/comprehension fixtures pass and the complete guard + file is green after the implementation. ## Next -- Route every probe through shared structure-based resolution, retaining - resolvable dict-merge entries alongside opaque sentinels. - Run the guard file, precision and benign batteries, full `populace-build` suite, repository Ruff, and `git diff --check`. - Restore `PROGRESS.md` exactly to `origin/main`, commit every coherent step diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index dd763bd5..9a7f7124 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -1182,6 +1182,8 @@ def _static_structure( for element in node.elts: if isinstance(element, ast.Starred): inner_value = _static_literal_value(element.value, constants) + if not isinstance(inner_value, (list, tuple)): + inner_value = _static_structure(element.value, constants) if isinstance(inner_value, (list, tuple)): resolved.extend(inner_value) else: @@ -1190,8 +1192,7 @@ def _static_structure( inner = _static_structure(element, constants) resolved.append(inner if inner is not None else _OPAQUE_STATIC_VALUE) else: - value = _static_literal_value(element, constants) - resolved.append(value) + resolved.append(_static_value_or_structure(element, constants)) return tuple(resolved) return None @@ -1208,16 +1209,25 @@ def _static_dict_entries( if isinstance(mapping, dict): return _StaticDictEntries(mapping.items()) if isinstance(node, ast.Dict): - if any(key is None for key in node.keys): - return None - entries = _StaticDictEntries( - ( - _static_value_or_structure(key, constants), - _static_value_or_structure(value, constants), - ) - for key, value in zip(node.keys, node.values, strict=True) - if key is not None - ) + resolved_entries: list[tuple[object, object]] = [] + for key, value in zip(node.keys, node.values, strict=True): + if key is not None: + resolved_entries.append( + ( + _static_value_or_structure(key, constants), + _static_value_or_structure(value, constants), + ) + ) + continue + expanded = _static_dict_entries(value, constants) + if expanded is None: + # An unknown **mapping may contribute or replace any key. + # Preserve that uncertainty without discarding entries + # resolved before or after it. + resolved_entries.append((_OPAQUE_STATIC_VALUE, _OPAQUE_STATIC_VALUE)) + else: + resolved_entries.extend(expanded) + entries = _StaticDictEntries(resolved_entries) elif ( isinstance(node, ast.Call) and isinstance(node.func, ast.Name) @@ -1299,33 +1309,6 @@ def _contains_opaque_static_value(value: object) -> bool: return False -def _static_dict_values_expression(node: ast.AST) -> ast.AST | None: - """Return only the syntactic value payload of a supported static dict.""" - - if isinstance(node, ast.Dict): - if any(key is None for key in node.keys): - return None - return ast.Tuple(elts=list(node.values), ctx=ast.Load()) - if ( - isinstance(node, ast.Call) - and isinstance(node.func, ast.Name) - and node.func.id == "dict" - and len(node.args) == 1 - and not node.keywords - ): - argument = node.args[0] - if isinstance(argument, ast.Dict): - return _static_dict_values_expression(argument) - if isinstance(argument, (ast.List, ast.Tuple, ast.Set)): - values: list[ast.AST] = [] - for entry in argument.elts: - if not isinstance(entry, (ast.List, ast.Tuple)) or len(entry.elts) != 2: - return None - values.append(entry.elts[1]) - return ast.Tuple(elts=values, ctx=ast.Load()) - return None - - def _value_bears_strings(value: object) -> bool: """True when a resolved value carries string material at any depth.""" @@ -1340,44 +1323,11 @@ def _value_bears_strings(value: object) -> bool: return False -def _static_container_expression(node: ast.AST) -> ast.AST | None: - """The literal-container core of an iterable expression, if any. - - Recognizes literal containers directly, static-dict ``.items()`` and - ``.values()``, and ``dict()``. Their literal content is - eligible for the conservative fragment fallback; ``.values()`` narrows - the core to values rather than mapping keys. Dynamic expressions that - merely mention strings (``mapping.get("person", {}).values()``) return - None so key lookups never trip the fragment rule.""" - - if isinstance(node, (ast.List, ast.Tuple, ast.Set, ast.Dict)): - return node - if ( - isinstance(node, ast.Call) - and isinstance(node.func, ast.Attribute) - and node.func.attr in {"items", "values"} - and not node.args - and not node.keywords - ): - if node.func.attr == "values": - return _static_dict_values_expression(node.func.value) - return _static_container_expression(node.func.value) - if ( - isinstance(node, ast.Call) - and isinstance(node.func, ast.Name) - and node.func.id == "dict" - and len(node.args) == 1 - and not node.keywords - ): - return _static_container_expression(node.args[0]) - return None - - def _static_iteration_value( node: ast.AST, constants: list[dict[str, object]], ) -> object: - """Resolve the supported static value iterated by a loop binder.""" + """Resolve the one structural value shared by binders and probes.""" if ( isinstance(node, ast.Call) @@ -1397,7 +1347,15 @@ def _static_iteration_value( value = _static_literal_value(node, constants) if isinstance(value, _StaticDictEntries): return tuple(key for key, _value in value) - return value + if value is not _OPAQUE_STATIC_VALUE: + return value + structure = _static_structure(node, constants) + if structure is not None: + return structure + entries = _static_dict_entries(node, constants) + if entries is not None: + return tuple(key for key, _value in entries) + return _OPAQUE_STATIC_VALUE def _static_iteration_string_choices( @@ -1415,38 +1373,10 @@ def _static_iteration_string_choices( return choices or None -def _iterable_carries_guarded_fragments(node: ast.AST) -> bool: - """True when a literal-container iterable holds string constants that - assemble into a guarded column name. Applies only to genuinely static - container expressions (see _static_container_expression) so dynamic - iterables that merely mention strings never trip it. Fragments shorter - than four characters are ignored.""" - - container = _static_container_expression(node) - if container is None: - return False - fragments = [ - constant.value - for constant in ast.walk(container) - if isinstance(constant, ast.Constant) - and isinstance(constant.value, str) - and len(constant.value) >= 4 - ] - return any( - fragment in column - for fragment in fragments - for column in _OPERATOR_SOURCE_COLUMNS - ) - - -def _resolved_value_carries_guarded_fragments( +def _structure_carries_guarded_fragments( node: ast.AST, constants: list[dict[str, object]] ) -> bool: - """Bound-value counterpart of the syntactic fragment probe. - - Names, structures, and supported static-dict views resolve to the value - actually iterated; its string leaves are checked for guarded fragments. - """ + """Leaf-walk the same resolved iterable structure used by the binder.""" value = _static_iteration_value(node, constants) if value is _OPAQUE_STATIC_VALUE: @@ -1468,6 +1398,22 @@ def _resolved_value_carries_guarded_fragments( ) +def _iterable_carries_guarded_fragments( + node: ast.AST, constants: list[dict[str, object]] +) -> bool: + """Inline-container fragment probe over the shared structural value.""" + + return _structure_carries_guarded_fragments(node, constants) + + +def _resolved_value_carries_guarded_fragments( + node: ast.AST, constants: list[dict[str, object]] +) -> bool: + """Bound-value fragment probe over the shared structural value.""" + + return _structure_carries_guarded_fragments(node, constants) + + class _SourceReadVisitor(ast.NodeVisitor): def __init__( self, @@ -1761,10 +1707,6 @@ def _bind_iteration_rows( """ literal = _static_iteration_value(iterable, self.constants) - if literal is _OPAQUE_STATIC_VALUE or not isinstance(literal, (list, tuple)): - structure = _static_structure(iterable, self.constants) - if structure is not None: - literal = structure if literal is _OPAQUE_STATIC_VALUE or not isinstance(literal, (list, tuple)): return False, False rows = tuple(literal) @@ -1821,6 +1763,11 @@ def _bind_column(element: ast.Name, values: tuple) -> None: # it precisely (sol #583 round 12). self._bind_iteration_target(element, None) self.constants[-1][element.id] = structured[0] + if _contains_opaque_static_value(structured[0]): + # Retaining the structure makes its known leaves + # available to later destructuring, but its opaque + # leaves still make this binding incomplete. + fully_propagated = False return def _leaves(value: object) -> tuple[str, ...]: @@ -2090,7 +2037,8 @@ def visit_For(self, node: ast.For) -> None: if values is None: values = _static_iteration_string_choices(node.iter, self.constants) carries_guarded_fragments = _iterable_carries_guarded_fragments( - node.iter + node.iter, + self.constants, ) or _resolved_value_carries_guarded_fragments(node.iter, self.constants) rows_recognized, fully_propagated = self._bind_iteration_rows( node.target, @@ -2166,7 +2114,8 @@ def _visit_comprehension( ) self._visit_access_target(generator.target) carries_guarded_fragments = _iterable_carries_guarded_fragments( - generator.iter + generator.iter, + self.constants, ) or _resolved_value_carries_guarded_fragments( generator.iter, self.constants, @@ -3781,9 +3730,9 @@ def f(dynamic): } for name, source in sources.items(): accesses = _source_spine_accesses(source) - assert any( - "unpropagatable target geometry" in access for access in accesses - ), name + assert any("unpropagatable target geometry" in access for access in accesses), ( + name + ) def test_starred_iterable_wrappers_fail_closed_for_star_targets(): @@ -3809,9 +3758,9 @@ def f(): """ for source in (loop, comprehension): accesses = _source_spine_accesses(source) - assert any( - "unpropagatable target geometry" in access for access in accesses - ), source + assert any("unpropagatable target geometry" in access for access in accesses), ( + source + ) def test_partial_static_dict_merges_retain_entries_and_incompleteness(): @@ -3838,9 +3787,9 @@ def f(dynamic): for source in (loop, comprehension): accesses = _source_spine_accesses(source) assert any("person_support_channel" in access for access in accesses), source - assert any( - "unpropagatable target geometry" in access for access in accesses - ), source + assert any("unpropagatable target geometry" in access for access in accesses), ( + source + ) def test_starred_static_rows_with_dynamic_tail_dual_report(): @@ -3867,9 +3816,9 @@ def f(dynamic): for source in (loop, comprehension): accesses = _source_spine_accesses(source) assert any("person_support_channel" in access for access in accesses), source - assert any( - "unpropagatable target geometry" in access for access in accesses - ), source + assert any("unpropagatable target geometry" in access for access in accesses), ( + source + ) def test_dict_items_and_starred_row_iteration_are_in_scope(): From e8abe853c85b3ce72be935b4da803e7578483727 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 05:38:04 -0400 Subject: [PATCH 80/94] Record round 13 validation receipts --- PROGRESS.md | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index fe712441..a25e9bdb 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,12 +2,11 @@ ## State -PR #583 round-13 fix lane is in progress on -`multispine-pool-build-578` from `6d903a6`. The review HOLD identifies one -resolver divergence: the loop fail-closed trigger and fragment/value probes do -not use the binder's structure-based resolution, so starred wrappers and -partially static dict merges can hide guarded fragments or suppress required -loop records. +PR #583 round-13 guard remediation is implemented and validated locally on +`multispine-pool-build-578` from `6d903a6`. The loop fail-closed trigger, +fragment probes, and binder now share structure-based iterable resolution, so +starred wrappers and partially static dict merges cannot hide guarded +fragments or suppress required loop records. ## Done @@ -29,10 +28,18 @@ loop records. retained partial nested structures while marking their bindings incomplete. - Confirmed all 15 new loop/comprehension fixtures pass and the complete guard file is green after the implementation. +- Confirmed zero direct guard findings in `acs_transfer.py`, + `capital_gain_details.py`, `housing_inputs.py`, and both congressional + district vintage modules; the focused graph/registry/benign battery passed + 23 tests. +- Passed the five focused runtime precision files: 90 passed, 4 skipped, with + three pre-existing warnings. +- Passed the full `populace-build` suite: 3,348 passed, 85 skipped, with five + pre-existing warnings. +- Passed repository-wide Ruff, guard-file format checking, and + `git diff --check`. ## Next -- Run the guard file, precision and benign batteries, full `populace-build` - suite, repository Ruff, and `git diff --check`. - Restore `PROGRESS.md` exactly to `origin/main`, commit every coherent step locally without pushing, and write `/private/tmp/583_fix6_handoff.md`. From 4c2e068be0a4bd138a352a1385e8981e11a8461f Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 05:38:52 -0400 Subject: [PATCH 81/94] Restore root progress journal --- PROGRESS.md | 73 ++++++++++++++++++++++++++++------------------------- 1 file changed, 38 insertions(+), 35 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index a25e9bdb..f3158fdd 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,44 +2,47 @@ ## State -PR #583 round-13 guard remediation is implemented and validated locally on -`multispine-pool-build-578` from `6d903a6`. The loop fail-closed trigger, -fragment probes, and binder now share structure-based iterable resolution, so -starred wrappers and partially static dict merges cannot hide guarded -fragments or suppress required loop records. +Populace #516 whole-row donor outlier screen is complete on +`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515 +interim carve merged as #525). The `puf_tax_detail` donor now drops tax units +whose grouped raw mortgage interest reaches $10M before the #515 carve +(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T +of phantom mortgage-interest mass), with the checkpoint schema bumped to v3 +so post-carve pre-screen checkpoints rebuild. ## Done -- Confirmed the requested branch, clean worktree, and exact starting commit. -- Read `CLAUDE.md`, the GitNexus debugging skill, and the round-13 review HOLD - at `/Users/maxghenis/PolicyEngine/_buildo-runtime/reviews/sol_583_r13.log`. -- Recorded the required shared structure resolution, partial-dict retention, - dual-report, fixture-backfill, precision, suite, lint, no-push, root-journal, - and handoff constraints. -- Recovered the exact nine previously silent round-12 nested-structure mirrors - from the authenticated reviewer transcript and committed them as explicit - fail-closed regression fixtures. -- Added the three exact round-13 loop repros and their comprehension mirrors, - pinning loop-only classification for the starred target and dual reports for - the partial dict merge and dynamic-tail starred rows. -- Unified the binder and both fragment probes on one structure-based iterable - resolver, including starred splicing and dict-entry views. -- Preserved resolvable partial-dict merge entries beside opaque sentinels and - retained partial nested structures while marking their bindings incomplete. -- Confirmed all 15 new loop/comprehension fixtures pass and the complete guard - file is green after the implementation. -- Confirmed zero direct guard findings in `acs_transfer.py`, - `capital_gain_details.py`, `housing_inputs.py`, and both congressional - district vintage modules; the focused graph/registry/benign battery passed - 23 tests. -- Passed the five focused runtime precision files: 90 passed, 4 skipped, with - three pre-existing warnings. -- Passed the full `populace-build` suite: 3,348 passed, 85 skipped, with five - pre-existing warnings. -- Passed repository-wide Ruff, guard-file format checking, and - `git diff --check`. +- Confirmed a clean starting worktree at `aef1c56`. +- Read the repository guidance and established the #515 donor carve as the + screen's required downstream boundary. +- Started source-level audits of every donor-frame consumer, checkpoint + validation, row-count pins, and existing donor-fact summaries. +- Attempted the requested GitNexus impact workflow; the managed filesystem + denied its global registry write. Its local index also exposed a broad + `build/` ignore mismatch, so the completed impact audit uses direct source + call sites and tests. +- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the + structural rationale and pinned-artifact receipts. +- Added a whole-row screen on grouped raw person `home_mortgage_interest` + after tax-unit assembly, before the #515 carve, with retained-index reset. +- Confirmed no downstream consumer pairs donor rows to the original HDF arrays + or carries a stale donor-length vector; values and weights always originate + from the same screened frame. +- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale + checkpoint regression track the live constant while retaining literal-v1 + corruptions. +- Added regression coverage for the exact grouped boundary, whole-row removal, + retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant. +- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets + 139; populace-data 138 with 1 skip. The directly affected tail-bound suite + adds 12 passes. Ruff format/check and `git diff --check` are clean. +- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line + audit, expected 208,611-row real-artifact effect, verification results, count + sweep, and deliberately untouched surfaces. ## Next -- Restore `PROGRESS.md` exactly to `origin/main`, commit every coherent step - locally without pushing, and write `/private/tmp/583_fix6_handoff.md`. +- PR #527 review cycle, then merge. After both #525 and #527: rebuild the + base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a + run that holds per `us_critical_targets.py`. +- Root record-level ETL carve stays open on populace#515. From 1340c0cd892d52721e924b21e03ba174b8c4e02f Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 05:57:17 -0400 Subject: [PATCH 82/94] Round 14: starred dict views route through the shared iteration resolver MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sol round-14: [*mapping.items()] wrappers bypassed the unification — the starred branch spliced only list/tuple operands, collapsing partial {**BASE, k: dynamic} merges to one opaque sentinel before any probe could see the retained entries. The starred branch now falls through to _static_iteration_value, the same path bare views take, so retained entries catch by name, dynamic siblings dual-report, fully static views catch cleanly, and fragment-free views stay silent. All four forms are committed fixtures. Co-Authored-By: Claude Fable 5 --- .../tests/test_us_spine_blindness.py | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 9a7f7124..34b9ac4f 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -1184,6 +1184,13 @@ def _static_structure( inner_value = _static_literal_value(element.value, constants) if not isinstance(inner_value, (list, tuple)): inner_value = _static_structure(element.value, constants) + if not isinstance(inner_value, (list, tuple)): + # Starred dict views ([*mapping.items()]) resolve + # through the SAME iteration resolver the binder uses, + # so partial merges keep their retained entries and + # opaque sentinels instead of collapsing + # (sol #583 round 14). + inner_value = _static_iteration_value(element.value, constants) if isinstance(inner_value, (list, tuple)): resolved.extend(inner_value) else: @@ -3610,6 +3617,58 @@ def f(df): assert not _source_spine_accesses(benign_bound) +def test_starred_dict_views_resolve_through_the_shared_iteration_path(): + """Sol #583 round 14: [*mapping.items()] wrappers — including partial + {**BASE, k: dynamic} merges — resolve through the same iteration + resolver as bare views, so retained entries catch by name and the + dynamic sibling dual-reports; fully static views catch by name alone; + fragment-free views stay clean.""" + + composed_partial = """ +BASE = {"person": "support_channel"} + + +def f(dynamic): + for entity, suffix in [*{**BASE, "state": dynamic}.items()]: + sink(f"{entity}_{suffix}") +""" + composed_comprehension = """ +BASE = {"person": "support_channel"} + + +def f(dynamic): + return [f"{e}_{sfx}" for e, sfx in [*{**BASE, "state": dynamic}.items()]] +""" + fully_static_view = """ +BASE = {"person": "support_channel"} + + +def f(): + for entity, suffix in [*BASE.items()]: + sink(f"{entity}_{suffix}") +""" + benign_view = """ +BASE = {"state": "fips"} + + +def f(): + for entity, suffix in [*BASE.items()]: + sink(f"{entity}_{suffix}") +""" + for source in (composed_partial, composed_comprehension): + accesses = _source_spine_accesses(source) + assert any("person_support_channel" in access for access in accesses), source + assert any("unpropagatable target geometry" in access for access in accesses), ( + source + ) + static_accesses = _source_spine_accesses(fully_static_view) + assert any("person_support_channel" in access for access in static_accesses) + assert not any( + "unpropagatable target geometry" in access for access in static_accesses + ) + assert _source_spine_accesses(benign_view) == () + + def test_round_11_and_12_string_material_never_reaches_opaque_bindings(): """Sol #583 rounds 11-12 closure fixtures, committed for provenance: stars swallowing strings, value-side starred splices, nested From d8554fd3178b17d564e6413f5baa6ba5ca1ad6ac Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 06:14:02 -0400 Subject: [PATCH 83/94] Round 15: dict(view) nestings share the resolver; partial scalars stay partial MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sol round-15's two sites: (1) dict(iterable) operands now fall through to the shared iteration resolver exactly like bare and starred views, so semantically identical view/merge/wrapper compositions classify identically at any nesting; (2) partial scalar enumerations return a _PartialStringChoices marker instead of silently dropping opaque siblings — consumers bind the known choices but the loop records the incompleteness, restoring the required dual report. Fragment-free partials stay clean. All three forms are committed fixtures. Co-Authored-By: Claude Fable 5 --- .../tests/test_us_spine_blindness.py | 63 ++++++++++++++++++- 1 file changed, 62 insertions(+), 1 deletion(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 34b9ac4f..5f32720d 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -514,6 +514,12 @@ def _static_string_shape( _OPAQUE_METHOD_ALIAS = ("", True) +class _PartialStringChoices(tuple): + """Known string members of a PARTIALLY static enumeration — opaque + siblings were dropped, so consumers bind the choices but must not + classify the binding complete (sol #583 round 15).""" + + class _StaticStringChoices(tuple): """Abstract alternatives bound by one static loop/comprehension target.""" @@ -1243,6 +1249,14 @@ def _static_dict_entries( and not node.keywords ): raw_entries = _static_value_or_structure(node.args[0], constants) + if not isinstance( + raw_entries, (_StaticDictEntries, dict, list, tuple, set, frozenset) + ): + # dict(view) nestings resolve through the SAME iteration + # resolver as bare views (sol #583 round 15). + iterated = _static_iteration_value(node.args[0], constants) + if isinstance(iterated, (list, tuple)): + raw_entries = iterated if isinstance(raw_entries, _StaticDictEntries): entries = raw_entries elif isinstance(raw_entries, dict): @@ -1377,6 +1391,10 @@ def _static_iteration_string_choices( if not value: return () choices = tuple(item for item in value if isinstance(item, str)) + if choices and len(choices) != len(value): + # Opaque siblings were dropped: the enumeration is PARTIAL, and + # the caller must not treat it as complete (sol #583 round 15). + return _PartialStringChoices(choices) return choices or None @@ -2058,7 +2076,11 @@ def visit_For(self, node: ast.For) -> None: node.target, values, ) - fully_propagated = values is not None and target_propagated + fully_propagated = ( + values is not None + and not isinstance(values, _PartialStringChoices) + and target_propagated + ) if not fully_propagated and carries_guarded_fragments: # Any refused or partial static binding over guarded fragments # fails closed at the loop, independently of scalar-string @@ -3617,6 +3639,45 @@ def f(df): assert not _source_spine_accesses(benign_bound) +def test_dict_view_nestings_and_partial_scalar_values_dual_report(): + """Sol #583 round 15: dict(view.items()).items() nestings resolve + through the shared iteration resolver, and partial scalar .values() + enumerations preserve their dropped-opacity so the loop dual-reports + beside the named catch; fragment-free partials stay clean.""" + + dict_view_nesting = """ +BASE = {"person": "support_channel"} + + +def f(dynamic): + for entity, suffix in dict({**BASE, "state": dynamic}.items()).items(): + sink(f"{entity}_{suffix}") +""" + partial_scalar_values = """ +BASE = {"known": "person"} + + +def f(dynamic): + for entity in {**BASE, "other": dynamic}.values(): + sink(f"{entity}_support_channel") +""" + benign_partial_values = """ +BASE = {"known": "state"} + + +def f(dynamic): + for entity in {**BASE, "other": dynamic}.values(): + sink(entity) +""" + for source in (dict_view_nesting, partial_scalar_values): + accesses = _source_spine_accesses(source) + assert any("person_support_channel" in access for access in accesses), source + assert any("unpropagatable target geometry" in access for access in accesses), ( + source + ) + assert _source_spine_accesses(benign_partial_values) == () + + def test_starred_dict_views_resolve_through_the_shared_iteration_path(): """Sol #583 round 14: [*mapping.items()] wrappers — including partial {**BASE, k: dynamic} merges — resolve through the same iteration From cc6b31eb1745b1d91af2b5a024993174fbdf9fc9 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 06:30:34 -0400 Subject: [PATCH 84/94] Round 16: comprehension layers share the resolver and keep partiality Sol round-16's two comprehension-path sites: the nested-comprehension evaluator now falls through to the shared iteration resolver (identity comprehensions over dict views classify exactly like the bare views), and comprehension binding checks the _PartialStringChoices marker so dropped opacity dual-reports through the layer. Fixtures committed with a fragment-free identity control. Co-Authored-By: Claude Fable 5 --- .../tests/test_us_spine_blindness.py | 54 ++++++++++++++++++- 1 file changed, 53 insertions(+), 1 deletion(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 5f32720d..595653ff 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -798,6 +798,14 @@ def _static_string_list( if generator.ifs or generator.is_async: return None values = _static_string_list(generator.iter, nested_constants) + if values is None: + # Shared iteration resolver: identity comprehensions over + # dict views classify like the bare views + # (sol #583 round 16). Partial enumerations propagate + # their marker so incompleteness survives the layer. + values = _static_iteration_string_choices( + generator.iter, nested_constants + ) if values is None: return None for name in _assigned_names(generator.target): @@ -2158,7 +2166,11 @@ def _visit_comprehension( generator.target, values, ) - fully_propagated = values is not None and target_propagated + fully_propagated = ( + values is not None + and not isinstance(values, _PartialStringChoices) + and target_propagated + ) if not fully_propagated and carries_guarded_fragments: self._record( generator.iter, @@ -3639,6 +3651,46 @@ def f(df): assert not _source_spine_accesses(benign_bound) +def test_identity_comprehensions_and_partial_comprehension_bindings(): + """Sol #583 round 16: identity comprehensions over dict views resolve + through the shared iteration path (classifying like the bare view), + and partial enumerations keep their marker through comprehension + binding so the dual report survives the layer.""" + + identity_comprehension = """ +VALUES = {"known": "person"} + + +def f(): + for entity in [value for value in VALUES.values()]: + sink(f"{entity}_support_channel") +""" + partial_comprehension = """ +BASE = {"known": "person"} + + +def f(dynamic): + return [ + f"{entity}_support_channel" + for entity in {**BASE, "other": dynamic}.values() + ] +""" + benign_identity = """ +VALUES = {"known": "state"} + + +def f(): + for entity in [value for value in VALUES.values()]: + sink(entity) +""" + identity_accesses = _source_spine_accesses(identity_comprehension) + assert any("person_support_channel" in a for a in identity_accesses) + partial_accesses = _source_spine_accesses(partial_comprehension) + assert any("person_support_channel" in a for a in partial_accesses) + assert any("unpropagatable target geometry" in a for a in partial_accesses) + assert _source_spine_accesses(benign_identity) == () + + def test_dict_view_nestings_and_partial_scalar_values_dual_report(): """Sol #583 round 15: dict(view.items()).items() nestings resolve through the shared iteration resolver, and partial scalar .values() From 6d9018e853c6fbab25956fc2004f1667c6a885da Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 06:45:50 -0400 Subject: [PATCH 85/94] Round 17: structural identity layers and partial sets classify Sol round-17: (1) structural identity comprehensions ([(e, s) for e, s in SOURCE]) resolve to their source rows inside the shared iteration resolver itself, so tuple-row identity layers classify exactly like the bare iterable at any depth; (2) starred set splices resolve through the literal -> structure -> iteration fall-through, keeping their members; (3) partial sets preserve known strings beside opaque sentinels, restoring the dual report. Fixtures committed with a fragment-free structural control. Co-Authored-By: Claude Fable 5 --- .../tests/test_us_spine_blindness.py | 119 +++++++++++++++++- 1 file changed, 116 insertions(+), 3 deletions(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 595653ff..d56b2566 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -557,10 +557,33 @@ def _static_literal_value( ): return node.value if isinstance(node, (ast.List, ast.Set, ast.Tuple)): - values = tuple( - _static_literal_value(element, constants) for element in node.elts - ) + spliced_elements: list[object] = [] + for element in node.elts: + if isinstance(element, ast.Starred): + # Starred expansions splice like the runtime, resolving + # through literal -> structure -> shared iteration paths + # so {*BASE.values()} keeps its members + # (sol #583 round 17). + inner = _static_literal_value(element.value, constants) + if not isinstance(inner, (list, tuple, set, frozenset)): + candidate = _static_iteration_value(element.value, constants) + if isinstance(candidate, (list, tuple)): + inner = candidate + if isinstance(inner, (list, tuple, set, frozenset)): + spliced_elements.extend(inner) + else: + spliced_elements.append(_OPAQUE_STATIC_VALUE) + else: + spliced_elements.append(_static_literal_value(element, constants)) + values = tuple(spliced_elements) if any(value is _OPAQUE_STATIC_VALUE for value in values): + if isinstance(node, ast.Set) and any( + isinstance(value, str) for value in values + ): + # Partial sets keep their known members beside opaque + # sentinels so string material stays visible and the + # partiality dual-reports (sol #583 round 17). + return tuple(values) return _OPAQUE_STATIC_VALUE if isinstance(node, ast.List): return list(values) @@ -792,6 +815,9 @@ def _static_string_list( return (*left, *right) return None if isinstance(node, (ast.ListComp, ast.SetComp, ast.GeneratorExp)): + structural_rows = _identity_structural_rows(node, constants) + if structural_rows is not None: + return structural_rows local: dict[str, object] = {} nested_constants = [*constants, local] for generator in node.generators: @@ -1358,6 +1384,11 @@ def _static_iteration_value( ) -> object: """Resolve the one structural value shared by binders and probes.""" + structural_rows = _identity_structural_rows(node, constants) + if structural_rows is not None: + # Structural identity comprehensions map rows to themselves, so + # they resolve to the source's rows (sol #583 round 17). + return structural_rows if ( isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) @@ -1431,6 +1462,44 @@ def _structure_carries_guarded_fragments( ) +def _identity_structural_rows( + node: ast.AST, constants: list[dict[str, object]] +) -> tuple | None: + """Rows of a STRUCTURAL identity comprehension. + + ``[(e, s) for e, s in SOURCE]`` (and set/generator forms) maps each + row to itself, so it resolves to SOURCE's rows through the shared + iteration resolver — tuple rows classify exactly like the bare + iterable (sol #583 round 17). Only pure identity elements qualify: + a Tuple/List elt whose names mirror the single generator's tuple + target in order, or a bare Name mirroring a name target. + """ + + if not isinstance(node, (ast.ListComp, ast.SetComp, ast.GeneratorExp)): + return None + if len(node.generators) != 1: + return None + generator = node.generators[0] + if generator.ifs or generator.is_async: + return None + target, elt = generator.target, node.elt + if isinstance(target, ast.Name) and isinstance(elt, ast.Name): + if target.id != elt.id: + return None + elif isinstance(target, (ast.Tuple, ast.List)) and isinstance( + elt, (ast.Tuple, ast.List) + ): + if len(target.elts) != len(elt.elts) or not all( + isinstance(t, ast.Name) and isinstance(e, ast.Name) and t.id == e.id + for t, e in zip(target.elts, elt.elts, strict=True) + ): + return None + else: + return None + rows = _static_iteration_value(generator.iter, constants) + return rows if isinstance(rows, (list, tuple)) else None + + def _iterable_carries_guarded_fragments( node: ast.AST, constants: list[dict[str, object]] ) -> bool: @@ -3651,6 +3720,50 @@ def f(df): assert not _source_spine_accesses(benign_bound) +def test_structural_identity_layers_and_partial_sets_classify(): + """Sol #583 round 17: structural identity comprehensions resolve to + their source rows through the shared iteration resolver, starred set + splices keep their members, and partial sets preserve known strings + beside opaque sentinels for the dual report.""" + + structural_identity = """ +BASE = {"person": "support_channel"} + + +def f(): + for entity, suffix in [(e, s) for e, s in BASE.items()]: + sink(f"{entity}_{suffix}") +""" + set_splice = """ +BASE = {"known": "person"} + + +def f(): + for entity in {*BASE.values()}: + sink(f"{entity}_support_channel") +""" + partial_set = """ +def f(dynamic): + for entity in {"person", dynamic}: + sink(f"{entity}_support_channel") +""" + benign_structural = """ +BASE = {"state": "fips"} + + +def f(): + for entity, suffix in [(e, s) for e, s in BASE.items()]: + sink(f"{entity}_{suffix}") +""" + for source in (structural_identity, set_splice): + accesses = _source_spine_accesses(source) + assert any("person_support_channel" in a for a in accesses), source + partial_accesses = _source_spine_accesses(partial_set) + assert any("person_support_channel" in a for a in partial_accesses) + assert any("unpropagatable target geometry" in a for a in partial_accesses) + assert _source_spine_accesses(benign_structural) == () + + def test_identity_comprehensions_and_partial_comprehension_bindings(): """Sol #583 round 16: identity comprehensions over dict views resolve through the shared iteration path (classifying like the bare view), From 0bad771a682bed54ad6a448922763034933d13e0 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 07:08:48 -0400 Subject: [PATCH 86/94] Round 18: identity dict comprehensions classify; repair the round-17 shortcut MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sol round-18 audit: (1) the round-17 identity shortcut returned structural rows from the STRING-list resolver, so partial rows reached a string-membership assumption and raised TypeError through identity layers — that resolver now returns only all-string tuples and defers structural rows to the iteration resolver, restoring the dual report; (2) identity dict comprehensions ({k: v for k, v in SOURCE}) now resolve through the identity resolver, and dict VIEWS over them route the same way, so .items()/.values() on an identity comprehension classify exactly like the bare source. Fixtures committed with fragment-free controls for both. Co-Authored-By: Claude Fable 5 --- .../tests/test_us_spine_blindness.py | 100 +++++++++++++++++- 1 file changed, 98 insertions(+), 2 deletions(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index d56b2566..1b0e219f 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -817,7 +817,13 @@ def _static_string_list( if isinstance(node, (ast.ListComp, ast.SetComp, ast.GeneratorExp)): structural_rows = _identity_structural_rows(node, constants) if structural_rows is not None: - return structural_rows + # This resolver's contract is a tuple of STRINGS; structural + # rows carrying non-string members (opaque sentinels, nested + # rows) belong to the iteration resolver, not here — returning + # them raised TypeError downstream (sol #583 round 18). + if all(isinstance(row, str) for row in structural_rows): + return tuple(structural_rows) + return None local: dict[str, object] = {} nested_constants = [*constants, local] for generator in node.generators: @@ -1398,6 +1404,19 @@ def _static_iteration_value( ): entries = _static_dict_entries(node.func.value, constants) if entries is None: + # {k: v for k, v in SOURCE}.items(): the identity dict + # comprehension's rows ARE the source's rows, so the view + # resolves through the identity resolver (sol #583 round 18). + identity_rows = _identity_structural_rows(node.func.value, constants) + if isinstance(identity_rows, (list, tuple)): + if node.func.attr == "items": + return tuple(identity_rows) + return tuple( + row[1] + if isinstance(row, (list, tuple)) and len(row) == 2 + else _OPAQUE_STATIC_VALUE + for row in identity_rows + ) return _OPAQUE_STATIC_VALUE return ( tuple(entries) @@ -1475,13 +1494,32 @@ def _identity_structural_rows( target in order, or a bare Name mirroring a name target. """ - if not isinstance(node, (ast.ListComp, ast.SetComp, ast.GeneratorExp)): + if not isinstance( + node, (ast.ListComp, ast.SetComp, ast.GeneratorExp, ast.DictComp) + ): return None if len(node.generators) != 1: return None generator = node.generators[0] if generator.ifs or generator.is_async: return None + if isinstance(node, ast.DictComp): + # {k: v for k, v in SOURCE} is an identity mapping when key and + # value mirror the tuple target in order — its rows ARE the + # source's rows (sol #583 round 18). + target = generator.target + if ( + not isinstance(target, (ast.Tuple, ast.List)) + or len(target.elts) != 2 + or not all(isinstance(element, ast.Name) for element in target.elts) + or not isinstance(node.key, ast.Name) + or not isinstance(node.value, ast.Name) + or node.key.id != target.elts[0].id + or node.value.id != target.elts[1].id + ): + return None + rows = _static_iteration_value(generator.iter, constants) + return rows if isinstance(rows, (list, tuple)) else None target, elt = generator.target, node.elt if isinstance(target, ast.Name) and isinstance(elt, ast.Name): if target.id != elt.id: @@ -3720,6 +3758,64 @@ def f(df): assert not _source_spine_accesses(benign_bound) +def test_dict_comprehension_identity_and_partial_layers_are_stable(): + """Sol #583 round 18: identity DICT comprehensions resolve like every + other identity form, and partial enumerations survive identity layers + with their dual report intact (a round-17 shortcut raised TypeError + there — this pins the repaired behavior).""" + + dict_identity = """ +DATA = {"person": "support_channel"} + + +def f(sink): + for entity, suffix in {e: s for e, s in DATA.items()}.items(): + sink(f"{entity}_{suffix}") +""" + dict_identity_values = """ +DATA = {"known": "person"} + + +def f(sink): + for entity in {k: v for k, v in DATA.items()}.values(): + sink(f"{entity}_support_channel") +""" + partial_identity_layer = """ +BASE = {"known": "person"} + + +def f(dynamic, sink): + DATA = {**BASE, "other": dynamic} + for entity in [value for value in DATA.values()]: + sink(f"{entity}_support_channel") +""" + benign_dict_identity = """ +DATA = {"state": "fips"} + + +def f(sink): + for entity, suffix in {e: s for e, s in DATA.items()}.items(): + sink(f"{entity}_{suffix}") +""" + benign_partial_layer = """ +BASE = {"known": "state"} + + +def f(dynamic, sink): + DATA = {**BASE, "other": dynamic} + for entity in [value for value in DATA.values()]: + sink(entity) +""" + for source in (dict_identity, dict_identity_values): + accesses = _source_spine_accesses(source) + assert any("person_support_channel" in a for a in accesses), source + partial_accesses = _source_spine_accesses(partial_identity_layer) + assert any("person_support_channel" in a for a in partial_accesses) + assert any("unpropagatable target geometry" in a for a in partial_accesses) + assert _source_spine_accesses(benign_dict_identity) == () + assert _source_spine_accesses(benign_partial_layer) == () + + def test_structural_identity_layers_and_partial_sets_classify(): """Sol #583 round 17: structural identity comprehensions resolve to their source rows through the shared iteration resolver, starred set From 944e77f19e87446cee87d89a8cc8d82c671b3329 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 07:31:32 -0400 Subject: [PATCH 87/94] Start PR 583 round 19 fix journal --- PROGRESS.md | 56 +++++++++++++++++------------------------------------ 1 file changed, 18 insertions(+), 38 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index f3158fdd..c2d71cfe 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,47 +2,27 @@ ## State -Populace #516 whole-row donor outlier screen is complete on -`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515 -interim carve merged as #525). The `puf_tax_detail` donor now drops tax units -whose grouped raw mortgage interest reaches $10M before the #515 carve -(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T -of phantom mortgage-interest mass), with the checkpoint schema bumped to v3 -so post-carve pre-screen checkpoints rebuild. +PR #583 round-19 guard remediation is in progress on +`multispine-pool-build-578` from clean commit `0bad771`. The audit isolates one +resolution-chain defect: single-hop bindings stop before the shared structural +iteration and dict-entry resolvers, so bound identity and partial layers can +classify differently from their inline equivalents. ## Done -- Confirmed a clean starting worktree at `aef1c56`. -- Read the repository guidance and established the #515 donor carve as the - screen's required downstream boundary. -- Started source-level audits of every donor-frame consumer, checkpoint - validation, row-count pins, and existing donor-fact summaries. -- Attempted the requested GitNexus impact workflow; the managed filesystem - denied its global registry write. Its local index also exposed a broad - `build/` ignore mismatch, so the completed impact audit uses direct source - call sites and tests. -- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the - structural rationale and pinned-artifact receipts. -- Added a whole-row screen on grouped raw person `home_mortgage_interest` - after tax-unit assembly, before the #515 carve, with retained-index reset. -- Confirmed no downstream consumer pairs donor rows to the original HDF arrays - or carries a stale donor-length vector; values and weights always originate - from the same screened frame. -- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale - checkpoint regression track the live constant while retaining literal-v1 - corruptions. -- Added regression coverage for the exact grouped boundary, whole-row removal, - retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant. -- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets - 139; populace-data 138 with 1 skip. The directly affected tail-bound suite - adds 12 passes. Ruff format/check and `git diff --check` are clean. -- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line - audit, expected 208,611-row real-artifact effect, verification results, count - sweep, and deliberately untouched surfaces. +- Confirmed the requested branch, clean starting worktree, and exact base + commit. +- Read the repository guidance and round-19 audit. +- Confirmed the four requested corrections and the acceptance invariant: + bound and inline forms must classify identically. +- Attempted the GitNexus debugging workflow; its graph tools are not exposed + in this workspace, so direct source/caller tracing is being used instead. ## Next -- PR #527 review cycle, then merge. After both #525 and #527: rebuild the - base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a - run that holds per `us_critical_targets.py`. -- Root record-level ETL carve stays open on populace#515. +- Add committed reviewer-reproduction fixtures and fragment-free controls. +- Repair binder fallthrough, mapping views over bound pair rows, partial-layer + propagation, and direct identity-dict-comprehension key iteration. +- Qualify the module contract to executable dataflow. +- Keep the full suite and ruff green at every commit, restore this journal to + `origin/main`, and write `/private/tmp/583_fix7_handoff.md` without pushing. From e8985c9133f7114314d81a30337a981c2be27974 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 07:42:47 -0400 Subject: [PATCH 88/94] Round 19: bound identity layers share iteration resolution Extend single-hop bindings through the shared iteration resolver, retain identity dict comprehensions as mapping entries, and derive keys for direct dict iteration. Pin bound/inline classification equivalence, partial dual reporting, mapping views, reviewer reproductions, and fragment-free controls; qualify literal coverage to executable dataflow. --- PROGRESS.md | 54 ++-- .../tests/test_us_spine_blindness.py | 286 +++++++++++++++++- 2 files changed, 318 insertions(+), 22 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index c2d71cfe..a5753d22 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,27 +2,45 @@ ## State -PR #583 round-19 guard remediation is in progress on -`multispine-pool-build-578` from clean commit `0bad771`. The audit isolates one -resolution-chain defect: single-hop bindings stop before the shared structural -iteration and dict-entry resolvers, so bound identity and partial layers can -classify differently from their inline equivalents. +PR #583 round-19 guard remediation is implemented and validated on +`multispine-pool-build-578`. Bound and inline identity/partial forms now share +one classification path: identity dictionaries retain mapping semantics, +direct dict iteration yields keys, and other supported structural iteration +results survive one name binding. ## Done -- Confirmed the requested branch, clean starting worktree, and exact base - commit. -- Read the repository guidance and round-19 audit. -- Confirmed the four requested corrections and the acceptance invariant: - bound and inline forms must classify identically. -- Attempted the GitNexus debugging workflow; its graph tools are not exposed - in this workspace, so direct source/caller tracing is being used instead. +- Confirmed the requested clean base `0bad771` and committed this live journal + before changing guard behavior. +- Traced all four audit findings to the binder cutoff after + `_static_string_list` and the context-free identity-`DictComp` row shortcut. +- Made identity dict comprehensions resolve as `_StaticDictEntries`, preserving + `.items()`/`.values()` across binding and runtime-correct key iteration both + inline and bound. +- Extended `_bind_name` through the shared iteration resolver without treating + opaque results as successful bindings or granting mapping APIs to ordinary + pair-row lists. +- Preserved known leaves and partiality through bound partial identity layers, + yielding the same named catch plus fail-closed records as the inline form. +- Qualified the literal-coverage contract to non-owner executable dataflow and + retained the pinned annotation/true-docstring exemptions. +- Added exact reviewer reproductions, bound/inline classification-multiset + comparisons, direct bound-key coverage, `.items()`/`.values()` coverage, + structural-row and partial-dict binding coverage, and fragment-free mirrors. +- Kept the fragile per-column partial-row regression and rounds 16-18 focused + set green: 5 passed. +- Guard file: 132 passed. Benign battery: 21 passed. Focused + registry/production/graph battery: 23 passed. +- Raw and governed scans are empty for `acs_transfer.py`, + `capital_gain_details.py`, `housing_inputs.py`, + `congressional_district_vintage.py`, and + `congressional_district_vintage_crosswalk.py`. +- Full workspace: 3,927 passed, 132 skipped, with seven known warnings. + Repository-wide ruff, guard-file format check, and `git diff --check` pass. ## Next -- Add committed reviewer-reproduction fixtures and fragment-free controls. -- Repair binder fallthrough, mapping views over bound pair rows, partial-layer - propagation, and direct identity-dict-comprehension key iteration. -- Qualify the module contract to executable dataflow. -- Keep the full suite and ruff green at every commit, restore this journal to - `origin/main`, and write `/private/tmp/583_fix7_handoff.md` without pushing. +- Commit the coherent round-19 implementation locally. +- Restore `PROGRESS.md` exactly to `origin/main`, validate that final journal + commit, and write `/private/tmp/583_fix7_handoff.md`. +- Do not push. diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 1b0e219f..90ef3686 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -18,7 +18,8 @@ refused or partial binding over a fragment-bearing supported static container fails closed at the iteration site; and - contraband guarded-name literals anywhere statically visible in non-owner - modules. + modules' executable dataflow; true docstrings and annotation forms are + deliberately exempt. Analysis is MODULE-LOCAL with single-hop name resolution. Three classes are out of scope by design, and naming them is the honest boundary. @@ -1281,6 +1282,18 @@ def _static_dict_entries( else: resolved_entries.extend(expanded) entries = _StaticDictEntries(resolved_entries) + elif isinstance(node, ast.DictComp): + # An identity dict comprehension is a MAPPING whose abstract entries + # are the source rows. Keeping that mapping identity here lets a bound + # result retain .items()/.values() semantics, while the iteration + # resolver remains free to derive runtime-correct keys for a bare + # iteration (sol #583 round 19). + identity_rows = _identity_structural_rows(node, constants) + if identity_rows is None or not all( + isinstance(row, (list, tuple)) and len(row) == 2 for row in identity_rows + ): + return None + entries = _StaticDictEntries((row[0], row[1]) for row in identity_rows) elif ( isinstance(node, ast.Call) and isinstance(node.func, ast.Name) @@ -1391,9 +1404,12 @@ def _static_iteration_value( """Resolve the one structural value shared by binders and probes.""" structural_rows = _identity_structural_rows(node, constants) - if structural_rows is not None: + if structural_rows is not None and not isinstance(node, ast.DictComp): # Structural identity comprehensions map rows to themselves, so - # they resolve to the source's rows (sol #583 round 17). + # list/set/generator forms resolve to the source's rows. Identity + # DICTS instead fall through to the entries resolver below: direct + # iteration then yields keys, exactly as it does at runtime + # (sol #583 rounds 17 and 19). return structural_rows if ( isinstance(node, ast.Call) @@ -1685,6 +1701,15 @@ def _bind_name( constant = _static_structure(value, self.constants) if constant is None: constant = _static_string_list(value, self.constants) + if constant is None: + iteration_value = _static_iteration_value(value, self.constants) + if iteration_value is not _OPAQUE_STATIC_VALUE: + # Comprehension results and other supported structural + # iterables must survive one binding exactly as their inline + # forms do. Dict comprehensions have already resolved to + # _StaticDictEntries above, retaining mapping-view semantics + # rather than degrading to untyped pair rows (round 19). + constant = iteration_value method_alias = self._method_alias(value) if method_alias is None and name in self.method_alias_history[scope_index]: method_alias = _OPAQUE_METHOD_ALIAS @@ -3608,7 +3633,7 @@ def f(df: pd.DataFrame, column="age"): def test_annotations_and_true_docstrings_are_not_dataflow() -> None: - """Every annotation form and each scope's real docstring are exempt.""" + """Annotations and real docstrings are exempt from executable dataflow.""" source = '''"""person_support_channel""" type Alias = Literal["person_support_channel"] @@ -3816,6 +3841,259 @@ def f(dynamic, sink): assert _source_spine_accesses(benign_partial_layer) == () +def test_round_19_bound_and_inline_iteration_classifications_are_identical(): + """Every round-19 reviewer repro matches its runtime-equivalent form. + + Comparison strips source locations but preserves duplicate reports, so a + bound layer cannot silently lose either a named catch or fail-closed + record. Each behavior also has a fragment-free mirror pinned clean. + """ + + def classifications(source: str) -> tuple[str, ...]: + return tuple( + sorted( + access.split(": ", maxsplit=1)[1] + for access in _source_spine_accesses(source) + ) + ) + + bare_identity_dict = """ +DATA = {"person": "support_channel"} + +for entity in {e: s for e, s in DATA.items()}: + sink(f"{entity}_support_channel") +""" + bare_dict = """ +DATA = {"person": "support_channel"} + +for entity in DATA: + sink(f"{entity}_support_channel") +""" + benign_bare_identity_dict = """ +DATA = {"state": "fips"} + +for entity in {e: s for e, s in DATA.items()}: + sink(f"{entity}_age") +""" + benign_bare_dict = """ +DATA = {"state": "fips"} + +for entity in DATA: + sink(f"{entity}_age") +""" + bound_copy_keys = """ +DATA = {"person": "support_channel"} +COPY = {e: s for e, s in DATA.items()} + +for entity in COPY: + sink(f"{entity}_support_channel") +""" + benign_bound_copy_keys = """ +DATA = {"state": "fips"} +COPY = {e: s for e, s in DATA.items()} + +for entity in COPY: + sink(f"{entity}_age") +""" + + bound_copy_items = """ +BASE = {"person": "support_channel"} +COPY = {e: s for e, s in BASE.items()} + +for entity, suffix in COPY.items(): + sink(f"{entity}_{suffix}") +""" + inline_copy_items = """ +BASE = {"person": "support_channel"} + +for entity, suffix in { + e: s for e, s in BASE.items() +}.items(): + sink(f"{entity}_{suffix}") +""" + benign_bound_copy_items = """ +BASE = {"state": "fips"} +COPY = {e: s for e, s in BASE.items()} + +for entity, suffix in COPY.items(): + sink(f"{entity}_{suffix}") +""" + benign_inline_copy_items = """ +BASE = {"state": "fips"} + +for entity, suffix in { + e: s for e, s in BASE.items() +}.items(): + sink(f"{entity}_{suffix}") +""" + + bound_copy_values = """ +BASE = {"known": "person"} +COPY = {e: s for e, s in BASE.items()} + +for entity in COPY.values(): + sink(f"{entity}_support_channel") +""" + inline_copy_values = """ +BASE = {"known": "person"} + +for entity in { + e: s for e, s in BASE.items() +}.values(): + sink(f"{entity}_support_channel") +""" + benign_bound_copy_values = """ +BASE = {"known": "state"} +COPY = {e: s for e, s in BASE.items()} + +for entity in COPY.values(): + sink(f"{entity}_age") +""" + benign_inline_copy_values = """ +BASE = {"known": "state"} + +for entity in { + e: s for e, s in BASE.items() +}.values(): + sink(f"{entity}_age") +""" + + bound_rows = """ +SOURCE = {"person": "support_channel"} +ROWS = [(entity, suffix) for entity, suffix in SOURCE.items()] + +for entity, suffix in ROWS: + sink(f"{entity}_{suffix}") +""" + inline_rows = """ +SOURCE = {"person": "support_channel"} + +for entity, suffix in [ + (key, value) for key, value in SOURCE.items() +]: + sink(f"{entity}_{suffix}") +""" + benign_bound_rows = """ +SOURCE = {"state": "fips"} +ROWS = [(entity, suffix) for entity, suffix in SOURCE.items()] + +for entity, suffix in ROWS: + sink(f"{entity}_{suffix}") +""" + benign_inline_rows = """ +SOURCE = {"state": "fips"} + +for entity, suffix in [ + (key, value) for key, value in SOURCE.items() +]: + sink(f"{entity}_{suffix}") +""" + + bound_partial_data = """ +BASE = {"known": "person"} + +def f(key, dynamic, sink): + DATA = {**BASE, key: dynamic} + for entity in DATA.values(): + sink(f"{entity}_support_channel") +""" + inline_partial_data = """ +BASE = {"known": "person"} + +def f(key, dynamic, sink): + for entity in {**BASE, key: dynamic}.values(): + sink(f"{entity}_support_channel") +""" + benign_bound_partial_data = """ +BASE = {"known": "state"} + +def f(key, dynamic, sink): + DATA = {**BASE, key: dynamic} + for entity in DATA.values(): + sink(f"{entity}_age") +""" + benign_inline_partial_data = """ +BASE = {"known": "state"} + +def f(key, dynamic, sink): + for entity in {**BASE, key: dynamic}.values(): + sink(f"{entity}_age") +""" + + bound_partial_layer = """ +BASE = {"known": "person"} + +def f(dynamic, sink): + DATA = {**BASE, "other": dynamic} + VALUES = [v for v in DATA.values()] + for entity in VALUES: + sink(f"{entity}_support_channel") +""" + inline_partial_layer = """ +BASE = {"known": "person"} + +def f(dynamic, sink): + DATA = {**BASE, "other": dynamic} + for entity in [v for v in DATA.values()]: + sink(f"{entity}_support_channel") +""" + benign_bound_partial_layer = """ +BASE = {"known": "state"} + +def f(dynamic, sink): + DATA = {**BASE, "other": dynamic} + VALUES = [v for v in DATA.values()] + for entity in VALUES: + sink(f"{entity}_age") +""" + benign_inline_partial_layer = """ +BASE = {"known": "state"} + +def f(dynamic, sink): + DATA = {**BASE, "other": dynamic} + for entity in [v for v in DATA.values()]: + sink(f"{entity}_age") +""" + + equivalent_pairs = ( + (bare_identity_dict, bare_dict), + (bound_copy_keys, bare_identity_dict), + (bound_copy_items, inline_copy_items), + (bound_copy_values, inline_copy_values), + (bound_rows, inline_rows), + (bound_partial_data, inline_partial_data), + (bound_partial_layer, inline_partial_layer), + ) + for bound_or_composed, runtime_equivalent in equivalent_pairs: + assert classifications(bound_or_composed) == classifications( + runtime_equivalent + ), bound_or_composed + + assert ( + classifications(bound_partial_layer).count( + "iteration over a static container carrying guarded-name fragments " + "with unpropagatable target geometry (fail-closed)" + ) + == 2 + ) + assert "contraband source column 'person_support_channel'" in classifications( + bound_partial_layer + ) + + fragment_free_pairs = ( + (benign_bare_identity_dict, benign_bare_dict), + (benign_bound_copy_keys, benign_bare_identity_dict), + (benign_bound_copy_items, benign_inline_copy_items), + (benign_bound_copy_values, benign_inline_copy_values), + (benign_bound_rows, benign_inline_rows), + (benign_bound_partial_data, benign_inline_partial_data), + (benign_bound_partial_layer, benign_inline_partial_layer), + ) + for bound_or_composed, runtime_equivalent in fragment_free_pairs: + assert _source_spine_accesses(bound_or_composed) == () + assert _source_spine_accesses(runtime_equivalent) == () + + def test_structural_identity_layers_and_partial_sets_classify(): """Sol #583 round 17: structural identity comprehensions resolve to their source rows through the shared iteration resolver, starred set From 75c82e96126e16e4a576b0506ff70a944ae452ee Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 07:45:30 -0400 Subject: [PATCH 89/94] Restore root progress journal --- PROGRESS.md | 74 +++++++++++++++++++++++++++-------------------------- 1 file changed, 38 insertions(+), 36 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index a5753d22..f3158fdd 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,45 +2,47 @@ ## State -PR #583 round-19 guard remediation is implemented and validated on -`multispine-pool-build-578`. Bound and inline identity/partial forms now share -one classification path: identity dictionaries retain mapping semantics, -direct dict iteration yields keys, and other supported structural iteration -results survive one name binding. +Populace #516 whole-row donor outlier screen is complete on +`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515 +interim carve merged as #525). The `puf_tax_detail` donor now drops tax units +whose grouped raw mortgage interest reaches $10M before the #515 carve +(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T +of phantom mortgage-interest mass), with the checkpoint schema bumped to v3 +so post-carve pre-screen checkpoints rebuild. ## Done -- Confirmed the requested clean base `0bad771` and committed this live journal - before changing guard behavior. -- Traced all four audit findings to the binder cutoff after - `_static_string_list` and the context-free identity-`DictComp` row shortcut. -- Made identity dict comprehensions resolve as `_StaticDictEntries`, preserving - `.items()`/`.values()` across binding and runtime-correct key iteration both - inline and bound. -- Extended `_bind_name` through the shared iteration resolver without treating - opaque results as successful bindings or granting mapping APIs to ordinary - pair-row lists. -- Preserved known leaves and partiality through bound partial identity layers, - yielding the same named catch plus fail-closed records as the inline form. -- Qualified the literal-coverage contract to non-owner executable dataflow and - retained the pinned annotation/true-docstring exemptions. -- Added exact reviewer reproductions, bound/inline classification-multiset - comparisons, direct bound-key coverage, `.items()`/`.values()` coverage, - structural-row and partial-dict binding coverage, and fragment-free mirrors. -- Kept the fragile per-column partial-row regression and rounds 16-18 focused - set green: 5 passed. -- Guard file: 132 passed. Benign battery: 21 passed. Focused - registry/production/graph battery: 23 passed. -- Raw and governed scans are empty for `acs_transfer.py`, - `capital_gain_details.py`, `housing_inputs.py`, - `congressional_district_vintage.py`, and - `congressional_district_vintage_crosswalk.py`. -- Full workspace: 3,927 passed, 132 skipped, with seven known warnings. - Repository-wide ruff, guard-file format check, and `git diff --check` pass. +- Confirmed a clean starting worktree at `aef1c56`. +- Read the repository guidance and established the #515 donor carve as the + screen's required downstream boundary. +- Started source-level audits of every donor-frame consumer, checkpoint + validation, row-count pins, and existing donor-fact summaries. +- Attempted the requested GitNexus impact workflow; the managed filesystem + denied its global registry write. Its local index also exposed a broad + `build/` ignore mismatch, so the completed impact audit uses direct source + call sites and tests. +- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the + structural rationale and pinned-artifact receipts. +- Added a whole-row screen on grouped raw person `home_mortgage_interest` + after tax-unit assembly, before the #515 carve, with retained-index reset. +- Confirmed no downstream consumer pairs donor rows to the original HDF arrays + or carries a stale donor-length vector; values and weights always originate + from the same screened frame. +- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale + checkpoint regression track the live constant while retaining literal-v1 + corruptions. +- Added regression coverage for the exact grouped boundary, whole-row removal, + retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant. +- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets + 139; populace-data 138 with 1 skip. The directly affected tail-bound suite + adds 12 passes. Ruff format/check and `git diff --check` are clean. +- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line + audit, expected 208,611-row real-artifact effect, verification results, count + sweep, and deliberately untouched surfaces. ## Next -- Commit the coherent round-19 implementation locally. -- Restore `PROGRESS.md` exactly to `origin/main`, validate that final journal - commit, and write `/private/tmp/583_fix7_handoff.md`. -- Do not push. +- PR #527 review cycle, then merge. After both #525 and #527: rebuild the + base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a + run that holds per `us_critical_targets.py`. +- Root record-level ETL carve stays open on populace#515. From 31d6a93a0f9a40b070a9a14c751a27bf4a3f32d1 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 08:52:48 -0400 Subject: [PATCH 90/94] Handle keys views, builtin wrappers, and filtered comprehensions --- .../tests/test_us_spine_blindness.py | 229 ++++++++++++++++-- 1 file changed, 212 insertions(+), 17 deletions(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 90ef3686..1c7cbb3c 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -13,10 +13,12 @@ static ``.replace`` chains, and the static-receiver case methods ``lower``, ``upper``, ``casefold``, ``title``, and ``capitalize``; - loop and comprehension propagation for supported literal or bound string - choices, structural list/tuple rows, and static-dict ``.items()`` and - ``.values()`` views (including supported ``dict(iterable)`` receivers); - refused or partial binding over a fragment-bearing supported static - container fails closed at the iteration site; and + choices, structural list/tuple rows, and static-dict ``.items()``, + ``.values()``, and ``.keys()`` views (including supported ``dict(iterable)`` + receivers); the element-preserving one-argument builtins ``list``, ``tuple``, + ``set``, ``frozenset``, ``sorted``, ``iter``, and ``reversed`` resolve through + that same iteration path; refused or partial binding over a fragment-bearing + supported static container fails closed at the iteration site; and - contraband guarded-name literals anywhere statically visible in non-owner modules' executable dataflow; true docstrings and annotation forms are deliberately exempt. @@ -37,10 +39,13 @@ Where multi-value loop bindings combine in one template, the guard may over-report combinations that no single row produces (a Cartesian over-catch). Over-reporting is the safe failure direction for a -tripwire; a module that trips it restructures its table. Partial dict -views likewise retain every entry with an opaque key because unknown -runtime keys may be distinct; if they collide, a value overwritten at -runtime can remain in the guard's conservative choice set. +tripwire; a module that trips it restructures its table. Filtered identity +comprehensions over supported static sources likewise resolve as their +unfiltered row sets: predicates can only remove rows, so retaining every row +is the sound over-approximation. Partial dict views likewise retain every entry +with an opaque key because unknown runtime keys may be distinct; if they +collide, a value overwritten at runtime can remain in the guard's conservative +choice set. """ from __future__ import annotations @@ -513,6 +518,9 @@ def _static_string_shape( _OPAQUE_STATIC_VALUE = object() _OPAQUE_STRING_PART = "\N{OBJECT REPLACEMENT CHARACTER}" _OPAQUE_METHOD_ALIAS = ("", True) +_ELEMENT_PRESERVING_ONE_ARGUMENT_BUILTINS = frozenset( + {"frozenset", "iter", "list", "reversed", "set", "sorted", "tuple"} +) class _PartialStringChoices(tuple): @@ -1326,6 +1334,13 @@ def _static_dict_entries( else: return None + if isinstance(node, ast.DictComp) and any( + generator.ifs for generator in node.generators + ): + # A predicate may remove a later duplicate key that would otherwise + # overwrite an earlier candidate. Retain the entire unfiltered row set + # so filtered identity mappings stay a sound over-approximation. + return entries if any(_contains_opaque_static_value(key) for key, _value in entries): # Unknown keys may or may not collide at runtime. Retain every row # rather than materializing them through one shared sentinel. @@ -1359,7 +1374,14 @@ def _static_dict_value( """Resolve a fully static dict or supported ``dict(iterable)``.""" entries = _static_dict_entries(node, constants) - if entries is None or _contains_opaque_static_value(entries): + if ( + entries is None + or _contains_opaque_static_value(entries) + or ( + isinstance(node, ast.DictComp) + and any(generator.ifs for generator in node.generators) + ) + ): return None try: return dict(entries) @@ -1403,6 +1425,17 @@ def _static_iteration_value( ) -> object: """Resolve the one structural value shared by binders and probes.""" + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id in _ELEMENT_PRESERVING_ONE_ARGUMENT_BUILTINS + and len(node.args) == 1 + and not node.keywords + ): + # Classification depends only on the iterated elements. These builtins + # materialize, reorder, or expose exactly those elements, so preserve + # the shared resolver's abstract value, including opaque sentinels. + return _static_iteration_value(node.args[0], constants) structural_rows = _identity_structural_rows(node, constants) if structural_rows is not None and not isinstance(node, ast.DictComp): # Structural identity comprehensions map rows to themselves, so @@ -1414,7 +1447,7 @@ def _static_iteration_value( if ( isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) - and node.func.attr in {"items", "values"} + and node.func.attr in {"items", "keys", "values"} and not node.args and not node.keywords ): @@ -1427,18 +1460,18 @@ def _static_iteration_value( if isinstance(identity_rows, (list, tuple)): if node.func.attr == "items": return tuple(identity_rows) + position = 0 if node.func.attr == "keys" else 1 return tuple( - row[1] + row[position] if isinstance(row, (list, tuple)) and len(row) == 2 else _OPAQUE_STATIC_VALUE for row in identity_rows ) return _OPAQUE_STATIC_VALUE - return ( - tuple(entries) - if node.func.attr == "items" - else tuple(value for _key, value in entries) - ) + if node.func.attr == "items": + return tuple(entries) + position = 0 if node.func.attr == "keys" else 1 + return tuple(entry[position] for entry in entries) value = _static_literal_value(node, constants) if isinstance(value, _StaticDictEntries): return tuple(key for key, _value in value) @@ -1517,7 +1550,7 @@ def _identity_structural_rows( if len(node.generators) != 1: return None generator = node.generators[0] - if generator.ifs or generator.is_async: + if generator.is_async: return None if isinstance(node, ast.DictComp): # {k: v for k, v in SOURCE} is an identity mapping when key and @@ -3841,6 +3874,168 @@ def f(dynamic, sink): assert _source_spine_accesses(benign_partial_layer) == () +def test_round_20_exact_keys_wrapper_and_filtered_repros_classify(): + """The round-20 reviewer constructions catch inline and after binding.""" + + hostile_sources = ( + """ +DATA = {"person": "support_channel"} +for entity in DATA.keys(): + sink(f"{entity}_support_channel") +""", + """ +DATA = {"person": "support_channel"} +KEYS = DATA.keys() +for entity in KEYS: + sink(f"{entity}_support_channel") +""", + """ +DATA = {"person": "support_channel"} +for entity, suffix in list(DATA.items()): + sink(f"{entity}_{suffix}") +""", + """ +DATA = {"person": "support_channel"} +ROWS = list(DATA.items()) +for entity, suffix in ROWS: + sink(f"{entity}_{suffix}") +""", + """ +DATA = {"person": "support_channel"} +for entity, suffix in sorted(DATA.items()): + sink(f"{entity}_{suffix}") +""", + """ +DATA = {"person": "support_channel"} +ROWS = sorted(DATA.items()) +for entity, suffix in ROWS: + sink(f"{entity}_{suffix}") +""", + """ +DATA = {"person": "support_channel", "state": "fips"} +for entity, suffix in [ + (e, s) for e, s in DATA.items() if e != "state" +]: + sink(f"{entity}_{suffix}") +""", + """ +DATA = {"person": "support_channel", "state": "fips"} +ROWS = [(e, s) for e, s in DATA.items() if e != "state"] +for entity, suffix in ROWS: + sink(f"{entity}_{suffix}") +""", + ) + benign_sources = ( + """ +DATA = {"state": "fips"} +for entity in DATA.keys(): + sink(f"{entity}_support_channel") +""", + """ +DATA = {"state": "fips"} +KEYS = DATA.keys() +for entity in KEYS: + sink(f"{entity}_support_channel") +""", + """ +DATA = {"state": "fips"} +for entity, suffix in list(DATA.items()): + sink(f"{entity}_{suffix}") +""", + """ +DATA = {"state": "fips"} +ROWS = list(DATA.items()) +for entity, suffix in ROWS: + sink(f"{entity}_{suffix}") +""", + """ +DATA = {"state": "fips"} +for entity, suffix in sorted(DATA.items()): + sink(f"{entity}_{suffix}") +""", + """ +DATA = {"state": "fips"} +ROWS = sorted(DATA.items()) +for entity, suffix in ROWS: + sink(f"{entity}_{suffix}") +""", + """ +DATA = {"state": "fips"} +for entity, suffix in [ + (e, s) for e, s in DATA.items() if e != "state" +]: + sink(f"{entity}_{suffix}") +""", + """ +DATA = {"state": "fips"} +ROWS = [(e, s) for e, s in DATA.items() if e != "state"] +for entity, suffix in ROWS: + sink(f"{entity}_{suffix}") +""", + ) + + for source in hostile_sources: + assert any( + "person_support_channel" in access + for access in _source_spine_accesses(source) + ), source + for source in benign_sources: + assert _source_spine_accesses(source) == (), source + + +def test_round_20_partial_keys_views_preserve_the_dual_report(): + """Known keys classify while an opaque sibling still fails closed.""" + + def classifications(source: str) -> tuple[str, ...]: + return tuple( + sorted( + access.split(": ", maxsplit=1)[1] + for access in _source_spine_accesses(source) + ) + ) + + bound = """ +BASE = {"person": "age"} +def f(key, dynamic, sink): + DATA = {**BASE, key: dynamic} + KEYS = DATA.keys() + for entity in KEYS: + sink(f"{entity}_support_channel") +""" + inline = """ +BASE = {"person": "age"} +def f(key, dynamic, sink): + DATA = {**BASE, key: dynamic} + for entity in DATA.keys(): + sink(f"{entity}_support_channel") +""" + benign_bound = """ +BASE = {"state": "fips"} +def f(key, dynamic, sink): + DATA = {**BASE, key: dynamic} + KEYS = DATA.keys() + for entity in KEYS: + sink(f"{entity}_age") +""" + benign_inline = """ +BASE = {"state": "fips"} +def f(key, dynamic, sink): + DATA = {**BASE, key: dynamic} + for entity in DATA.keys(): + sink(f"{entity}_age") +""" + + expected = ( + "contraband source column 'person_support_channel'", + "iteration over a static container carrying guarded-name fragments " + "with unpropagatable target geometry (fail-closed)", + ) + assert classifications(bound) == expected + assert classifications(inline) == expected + assert _source_spine_accesses(benign_bound) == () + assert _source_spine_accesses(benign_inline) == () + + def test_round_19_bound_and_inline_iteration_classifications_are_identical(): """Every round-19 reviewer repro matches its runtime-equivalent form. From f086fcdcc4879eab4aebe13ac6c401ec76328aec Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 08:59:43 -0400 Subject: [PATCH 91/94] Add round 20 iteration family regression matrices --- .../tests/test_us_spine_blindness.py | 494 +++++++++++++++++- 1 file changed, 474 insertions(+), 20 deletions(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 1c7cbb3c..1240f8c9 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -15,10 +15,11 @@ - loop and comprehension propagation for supported literal or bound string choices, structural list/tuple rows, and static-dict ``.items()``, ``.values()``, and ``.keys()`` views (including supported ``dict(iterable)`` - receivers); the element-preserving one-argument builtins ``list``, ``tuple``, - ``set``, ``frozenset``, ``sorted``, ``iter``, and ``reversed`` resolve through - that same iteration path; refused or partial binding over a fragment-bearing - supported static container fails closed at the iteration site; and + receivers); one layer of the element-preserving one-argument builtins + ``list``, ``tuple``, ``set``, ``frozenset``, ``sorted``, ``iter``, and + ``reversed`` resolves through that same iteration path; refused or partial + binding over a fragment-bearing supported static container fails closed at + the iteration site; and - contraband guarded-name literals anywhere statically visible in non-owner modules' executable dataflow; true docstrings and annotation forms are deliberately exempt. @@ -1475,6 +1476,11 @@ def _static_iteration_value( value = _static_literal_value(node, constants) if isinstance(value, _StaticDictEntries): return tuple(key for key, _value in value) + if isinstance(value, dict): + # Iteration over a mapping projects keys. Normalize here so an inline + # element-preserving wrapper and a wrapper bound before iteration take + # the same shared path (sol #583 rounds 19-20). + return tuple(value) if value is not _OPAQUE_STATIC_VALUE: return value structure = _static_structure(node, constants) @@ -3874,6 +3880,24 @@ def f(dynamic, sink): assert _source_spine_accesses(benign_partial_layer) == () +_ROUND_20_NAMED_FINDING = "contraband source column 'person_support_channel'" +_ROUND_20_FAIL_CLOSED_FINDING = ( + "iteration over a static container carrying guarded-name fragments " + "with unpropagatable target geometry (fail-closed)" +) + + +def _finding_classifications(source: str) -> tuple[str, ...]: + """Finding descriptions without locations, retaining duplicate reports.""" + + return tuple( + sorted( + access.split(": ", maxsplit=1)[1] + for access in _source_spine_accesses(source) + ) + ) + + def test_round_20_exact_keys_wrapper_and_filtered_repros_classify(): """The round-20 reviewer constructions catch inline and after binding.""" @@ -3986,14 +4010,6 @@ def test_round_20_exact_keys_wrapper_and_filtered_repros_classify(): def test_round_20_partial_keys_views_preserve_the_dual_report(): """Known keys classify while an opaque sibling still fails closed.""" - def classifications(source: str) -> tuple[str, ...]: - return tuple( - sorted( - access.split(": ", maxsplit=1)[1] - for access in _source_spine_accesses(source) - ) - ) - bound = """ BASE = {"person": "age"} def f(key, dynamic, sink): @@ -4025,23 +4041,376 @@ def f(key, dynamic, sink): sink(f"{entity}_age") """ - expected = ( - "contraband source column 'person_support_channel'", - "iteration over a static container carrying guarded-name fragments " - "with unpropagatable target geometry (fail-closed)", - ) - assert classifications(bound) == expected - assert classifications(inline) == expected + expected = (_ROUND_20_NAMED_FINDING, _ROUND_20_FAIL_CLOSED_FINDING) + assert _finding_classifications(bound) == expected + assert _finding_classifications(inline) == expected assert _source_spine_accesses(benign_bound) == () assert _source_spine_accesses(benign_inline) == () +@pytest.mark.parametrize("partial", [False, True], ids=["full", "partial"]) +@pytest.mark.parametrize( + "hostile", + [pytest.param(True, id="hostile"), pytest.param(False, id="benign")], +) +def test_round_20_keys_views_match_bare_dict_iteration( + partial: bool, + hostile: bool, +) -> None: + """Keys project keys, including projection-sensitive benign controls.""" + + known_key = "person" if hostile else "state" + # The benign value is deliberately hostile as a key projection control. + known_value = "age" if hostile else "person" + if partial: + setup = f'BASE = {{"{known_key}": "{known_value}"}}' + + def source(iterable: str, *, bind_view: bool = False) -> str: + lines = [setup, "def f(key, dynamic, sink):"] + lines.append(" DATA = {**BASE, key: dynamic}") + if bind_view: + lines.append(" KEYS = DATA.keys()") + lines.extend( + ( + f" for entity in {iterable}:", + ' sink(f"{entity}_support_channel")', + ) + ) + return "\n".join(lines) + + else: + setup = f'DATA = {{"{known_key}": "{known_value}"}}' + + def source(iterable: str, *, bind_view: bool = False) -> str: + lines = [setup, "def f(sink):"] + if bind_view: + lines.append(" KEYS = DATA.keys()") + lines.extend( + ( + f" for entity in {iterable}:", + ' sink(f"{entity}_support_channel")', + ) + ) + return "\n".join(lines) + + inline = _finding_classifications(source("DATA.keys()")) + bound = _finding_classifications(source("KEYS", bind_view=True)) + bare = _finding_classifications(source("DATA")) + expected = ( + (_ROUND_20_NAMED_FINDING, _ROUND_20_FAIL_CLOSED_FINDING) + if hostile and partial + else (_ROUND_20_NAMED_FINDING,) + if hostile + else () + ) + assert inline == bound == bare == expected + + +@pytest.mark.parametrize( + "wrapper", + sorted(_ELEMENT_PRESERVING_ONE_ARGUMENT_BUILTINS), +) +@pytest.mark.parametrize( + ("operand_case", "target", "sink_statement"), + ( + pytest.param( + "bare_dict", + "entity", + 'sink(f"{entity}_support_channel")', + id="bare-dict", + ), + pytest.param( + "items", + "entity, suffix", + 'sink(f"{entity}_{suffix}")', + id="items-view", + ), + pytest.param( + "values", + "entity", + 'sink(f"{entity}_support_channel")', + id="values-view", + ), + pytest.param( + "keys", + "entity", + 'sink(f"{entity}_support_channel")', + id="keys-view", + ), + pytest.param( + "partial_dict", + "entity", + 'sink(f"{entity}_support_channel")', + id="partial-dict", + ), + ), +) +@pytest.mark.parametrize( + "hostile", + [pytest.param(True, id="hostile"), pytest.param(False, id="benign")], +) +def test_round_20_element_preserving_wrapper_view_matrix( + wrapper: str, + operand_case: str, + target: str, + sink_statement: str, + hostile: bool, +) -> None: + """All seven wrappers classify like the same operand without a wrapper.""" + + if operand_case in {"bare_dict", "keys"}: + key = "person" if hostile else "state" + # A hostile benign-side value proves that only keys are projected. + value = "age" if hostile else "person" + setup = f'DATA = {{"{key}": "{value}"}}' + operand = "DATA" if operand_case == "bare_dict" else "DATA.keys()" + elif operand_case == "items": + key = "person" if hostile else "state" + value = "support_channel" if hostile else "fips" + setup = f'DATA = {{"{key}": "{value}"}}' + operand = "DATA.items()" + elif operand_case == "values": + key = "known" if hostile else "person" + value = "person" if hostile else "state" + setup = f'DATA = {{"{key}": "{value}"}}' + operand = "DATA.values()" + else: + key = "person" if hostile else "state" + # As above, the benign value catches an accidental values projection. + value = "age" if hostile else "person" + setup = f'BASE = {{"{key}": "{value}"}}' + operand = "{**BASE, key: dynamic}" + + def source(mode: str) -> str: + lines = [setup, "def f(key, dynamic, sink):"] + if mode == "bound": + lines.append(f" WRAPPED = {wrapper}({operand})") + iterable = "WRAPPED" + elif mode == "inline": + iterable = f"{wrapper}({operand})" + else: + iterable = operand + lines.extend( + ( + f" for {target} in {iterable}:", + f" {sink_statement}", + ) + ) + return "\n".join(lines) + + expected = ( + (_ROUND_20_NAMED_FINDING, _ROUND_20_FAIL_CLOSED_FINDING) + if hostile and operand_case == "partial_dict" + else (_ROUND_20_NAMED_FINDING,) + if hostile + else () + ) + assert _finding_classifications(source("bound")) == expected + assert _finding_classifications(source("inline")) == expected + assert _finding_classifications(source("unwrapped")) == expected + + +@pytest.mark.parametrize( + "wrapper", + sorted(_ELEMENT_PRESERVING_ONE_ARGUMENT_BUILTINS), +) +@pytest.mark.parametrize("partial", [False, True], ids=["full", "partial"]) +@pytest.mark.parametrize( + "hostile", + [pytest.param(True, id="hostile"), pytest.param(False, id="benign")], +) +def test_round_20_wrappers_preserve_static_structural_rows( + wrapper: str, + partial: bool, + hostile: bool, +) -> None: + """Opaque-with-fragments operands never collapse silently to empty.""" + + if hostile: + rows = ( + '(("person", "support_channel"), ("state", dynamic))' + if partial + else '(("person", "support_channel"), ("state", "fips"))' + ) + else: + rows = ( + '(("state", "fips"), ("county", dynamic))' + if partial + else '(("state", "fips"), ("county", "code"))' + ) + + def source(mode: str) -> str: + lines = ["def f(dynamic, sink):"] + if mode == "bound": + lines.extend((f" ROWS = {rows}", f" WRAPPED = {wrapper}(ROWS)")) + iterable = "WRAPPED" + elif mode == "inline": + iterable = f"{wrapper}({rows})" + else: + iterable = rows + lines.extend( + ( + f" for entity, suffix in {iterable}:", + ' sink(f"{entity}_{suffix}")', + ) + ) + return "\n".join(lines) + + expected = ( + (_ROUND_20_NAMED_FINDING, _ROUND_20_FAIL_CLOSED_FINDING) + if hostile and partial + else (_ROUND_20_NAMED_FINDING,) + if hostile + else () + ) + assert _finding_classifications(source("bound")) == expected + assert _finding_classifications(source("inline")) == expected + assert _finding_classifications(source("unwrapped")) == expected + + +def _round_20_identity_pair_comprehension( + kind: str, + iterable: str, + *, + filtered: bool, +) -> str: + clause = f"for e, s in {iterable}" + if filtered: + clause += ' if e != "state"' + if kind == "list": + return f"[(e, s) {clause}]" + if kind == "set": + return f"{{(e, s) {clause}}}" + if kind == "generator": + return f"((e, s) {clause})" + return f"{{e: s {clause}}}" + + +@pytest.mark.parametrize("kind", ("list", "set", "generator", "dict")) +@pytest.mark.parametrize("partial", [False, True], ids=["full", "partial"]) +@pytest.mark.parametrize( + "hostile", + [pytest.param(True, id="hostile"), pytest.param(False, id="benign")], +) +def test_round_20_filtered_identity_comprehension_matrix( + kind: str, + partial: bool, + hostile: bool, +) -> None: + """Every filtered identity form classifies as its unfiltered row set.""" + + if partial: + key = "person" if hostile else "state" + value = "support_channel" if hostile else "fips" + setup = f'BASE = {{"{key}": "{value}"}}' + else: + setup = ( + 'DATA = {"person": "support_channel", "state": "fips"}' + if hostile + else 'DATA = {"state": "fips"}' + ) + + def source(*, filtered: bool, bound: bool) -> str: + lines = [setup, "def f(key, dynamic, sink):"] + if partial: + lines.append(" DATA = {**BASE, key: dynamic}") + expression = _round_20_identity_pair_comprehension( + kind, + "DATA.items()", + filtered=filtered, + ) + if bound: + lines.append(f" ROWS = {expression}") + iterable = "ROWS.items()" if kind == "dict" else "ROWS" + else: + iterable = f"({expression}).items()" if kind == "dict" else expression + lines.extend( + ( + f" for entity, suffix in {iterable}:", + ' sink(f"{entity}_{suffix}")', + ) + ) + return "\n".join(lines) + + expected = ( + ( + _ROUND_20_NAMED_FINDING, + _ROUND_20_FAIL_CLOSED_FINDING, + _ROUND_20_FAIL_CLOSED_FINDING, + ) + if hostile and partial + else (_ROUND_20_NAMED_FINDING,) + if hostile + else () + ) + variants = ( + source(filtered=True, bound=True), + source(filtered=True, bound=False), + source(filtered=False, bound=True), + source(filtered=False, bound=False), + ) + for variant in variants: + assert _finding_classifications(variant) == expected, variant + + +@pytest.mark.parametrize( + "hostile", + [pytest.param(True, id="hostile"), pytest.param(False, id="benign")], +) +def test_round_20_filtered_identity_dict_retains_duplicate_key_candidates( + hostile: bool, +) -> None: + """A filter may remove a later duplicate, so every source row survives.""" + + key = "person" if hostile else "state" + first_value = "support_channel" if hostile else "fips" + setup = f'DATA = (("{key}", "{first_value}"), ("{key}", "age"))' + bound = f""" +{setup} +ROWS = {{entity: suffix for entity, suffix in DATA if suffix != "age"}} +for entity, suffix in ROWS.items(): + sink(f"{{entity}}_{{suffix}}") +""" + inline = f""" +{setup} +for entity, suffix in ( + {{entity: suffix for entity, suffix in DATA if suffix != "age"}} +).items(): + sink(f"{{entity}}_{{suffix}}") +""" + expected = (_ROUND_20_NAMED_FINDING,) if hostile else () + assert _finding_classifications(bound) == expected + assert _finding_classifications(inline) == expected + + +@pytest.mark.parametrize( + "module_name", + ( + "acs_transfer.py", + "capital_gain_details.py", + "housing_inputs.py", + "congressional_district_vintage.py", + "congressional_district_vintage_crosswalk.py", + ), +) +def test_round_20_named_production_modules_scan_clean_raw_and_governed( + module_name: str, +) -> None: + """The five requested production controls remain clean on both paths.""" + + source = (_US_RUNTIME / module_name).read_text(encoding="utf-8") + assert _source_spine_accesses(source) == () + assert _non_owner_source_spine_accesses(module_name, source) == () + + def test_round_19_bound_and_inline_iteration_classifications_are_identical(): """Every round-19 reviewer repro matches its runtime-equivalent form. Comparison strips source locations but preserves duplicate reports, so a bound layer cannot silently lose either a named catch or fail-closed - record. Each behavior also has a fragment-free mirror pinned clean. + record. Each behavior also has a fragment-free mirror pinned clean. The + round-20 keys, builtin-wrapper, and filtered-identity families extend the + same equivalence property. """ def classifications(source: str) -> tuple[str, ...]: @@ -4250,6 +4619,85 @@ def f(dynamic, sink): sink(f"{entity}_age") """ + bound_keys_view = """ +DATA = {"person": "age"} +KEYS = DATA.keys() +for entity in KEYS: + sink(f"{entity}_support_channel") +""" + inline_keys_view = """ +DATA = {"person": "age"} +for entity in DATA.keys(): + sink(f"{entity}_support_channel") +""" + benign_bound_keys_view = """ +DATA = {"state": "person"} +KEYS = DATA.keys() +for entity in KEYS: + sink(f"{entity}_support_channel") +""" + benign_inline_keys_view = """ +DATA = {"state": "person"} +for entity in DATA.keys(): + sink(f"{entity}_support_channel") +""" + + bound_filtered_rows = """ +DATA = {"person": "support_channel", "state": "fips"} +ROWS = [(e, s) for e, s in DATA.items() if e != "state"] +for entity, suffix in ROWS: + sink(f"{entity}_{suffix}") +""" + inline_filtered_rows = """ +DATA = {"person": "support_channel", "state": "fips"} +for entity, suffix in [(e, s) for e, s in DATA.items() if e != "state"]: + sink(f"{entity}_{suffix}") +""" + benign_bound_filtered_rows = """ +DATA = {"state": "fips"} +ROWS = [(e, s) for e, s in DATA.items() if e != "state"] +for entity, suffix in ROWS: + sink(f"{entity}_{suffix}") +""" + benign_inline_filtered_rows = """ +DATA = {"state": "fips"} +for entity, suffix in [(e, s) for e, s in DATA.items() if e != "state"]: + sink(f"{entity}_{suffix}") +""" + + wrapper_equivalent_pairs = tuple( + ( + f""" +DATA = {{"person": "support_channel"}} +ROWS = {wrapper}(DATA.items()) +for entity, suffix in ROWS: + sink(f"{{entity}}_{{suffix}}") +""", + f""" +DATA = {{"person": "support_channel"}} +for entity, suffix in {wrapper}(DATA.items()): + sink(f"{{entity}}_{{suffix}}") +""", + ) + for wrapper in sorted(_ELEMENT_PRESERVING_ONE_ARGUMENT_BUILTINS) + ) + benign_wrapper_equivalent_pairs = tuple( + ( + f""" +DATA = {{"state": "fips"}} +ROWS = {wrapper}(DATA.items()) +for entity, suffix in ROWS: + sink(f"{{entity}}_{{suffix}}") +""", + f""" +DATA = {{"state": "fips"}} +for entity, suffix in {wrapper}(DATA.items()): + sink(f"{{entity}}_{{suffix}}") +""", + ) + for wrapper in sorted(_ELEMENT_PRESERVING_ONE_ARGUMENT_BUILTINS) + ) + equivalent_pairs = ( (bare_identity_dict, bare_dict), (bound_copy_keys, bare_identity_dict), @@ -4258,6 +4706,9 @@ def f(dynamic, sink): (bound_rows, inline_rows), (bound_partial_data, inline_partial_data), (bound_partial_layer, inline_partial_layer), + (bound_keys_view, inline_keys_view), + (bound_filtered_rows, inline_filtered_rows), + *wrapper_equivalent_pairs, ) for bound_or_composed, runtime_equivalent in equivalent_pairs: assert classifications(bound_or_composed) == classifications( @@ -4283,6 +4734,9 @@ def f(dynamic, sink): (benign_bound_rows, benign_inline_rows), (benign_bound_partial_data, benign_inline_partial_data), (benign_bound_partial_layer, benign_inline_partial_layer), + (benign_bound_keys_view, benign_inline_keys_view), + (benign_bound_filtered_rows, benign_inline_filtered_rows), + *benign_wrapper_equivalent_pairs, ) for bound_or_composed, runtime_equivalent in fragment_free_pairs: assert _source_spine_accesses(bound_or_composed) == () From c978800659958738caa1f5b8d2be44d2647cec51 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 09:02:03 -0400 Subject: [PATCH 92/94] Strengthen wrapper table and filtered overcatch tests --- .../tests/test_us_spine_blindness.py | 23 +++++++++++++------ 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 1240f8c9..7b8d15cb 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -3885,6 +3885,15 @@ def f(dynamic, sink): "iteration over a static container carrying guarded-name fragments " "with unpropagatable target geometry (fail-closed)" ) +_ROUND_20_ELEMENT_PRESERVING_BUILTINS = ( + "list", + "tuple", + "set", + "frozenset", + "sorted", + "iter", + "reversed", +) def _finding_classifications(source: str) -> tuple[str, ...]: @@ -4108,7 +4117,7 @@ def source(iterable: str, *, bind_view: bool = False) -> str: @pytest.mark.parametrize( "wrapper", - sorted(_ELEMENT_PRESERVING_ONE_ARGUMENT_BUILTINS), + _ROUND_20_ELEMENT_PRESERVING_BUILTINS, ) @pytest.mark.parametrize( ("operand_case", "target", "sink_statement"), @@ -4212,7 +4221,7 @@ def source(mode: str) -> str: @pytest.mark.parametrize( "wrapper", - sorted(_ELEMENT_PRESERVING_ONE_ARGUMENT_BUILTINS), + _ROUND_20_ELEMENT_PRESERVING_BUILTINS, ) @pytest.mark.parametrize("partial", [False, True], ids=["full", "partial"]) @pytest.mark.parametrize( @@ -4360,21 +4369,21 @@ def source(*, filtered: bool, bound: bool) -> str: def test_round_20_filtered_identity_dict_retains_duplicate_key_candidates( hostile: bool, ) -> None: - """A filter may remove a later duplicate, so every source row survives.""" + """Even a hostile row removed at runtime stays in the over-approximation.""" key = "person" if hostile else "state" first_value = "support_channel" if hostile else "fips" setup = f'DATA = (("{key}", "{first_value}"), ("{key}", "age"))' bound = f""" {setup} -ROWS = {{entity: suffix for entity, suffix in DATA if suffix != "age"}} +ROWS = {{entity: suffix for entity, suffix in DATA if suffix == "age"}} for entity, suffix in ROWS.items(): sink(f"{{entity}}_{{suffix}}") """ inline = f""" {setup} for entity, suffix in ( - {{entity: suffix for entity, suffix in DATA if suffix != "age"}} + {{entity: suffix for entity, suffix in DATA if suffix == "age"}} ).items(): sink(f"{{entity}}_{{suffix}}") """ @@ -4679,7 +4688,7 @@ def f(dynamic, sink): sink(f"{{entity}}_{{suffix}}") """, ) - for wrapper in sorted(_ELEMENT_PRESERVING_ONE_ARGUMENT_BUILTINS) + for wrapper in _ROUND_20_ELEMENT_PRESERVING_BUILTINS ) benign_wrapper_equivalent_pairs = tuple( ( @@ -4695,7 +4704,7 @@ def f(dynamic, sink): sink(f"{{entity}}_{{suffix}}") """, ) - for wrapper in sorted(_ELEMENT_PRESERVING_ONE_ARGUMENT_BUILTINS) + for wrapper in _ROUND_20_ELEMENT_PRESERVING_BUILTINS ) equivalent_pairs = ( From d0de88919bd81ce820cb094d7272ffe4bf7993c1 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 09:36:46 -0400 Subject: [PATCH 93/94] Classify iteration wrappers by the membership principle --- .../tests/test_us_spine_blindness.py | 226 +++++++++++++++--- 1 file changed, 191 insertions(+), 35 deletions(-) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 7b8d15cb..6b0faf77 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -15,9 +15,12 @@ - loop and comprehension propagation for supported literal or bound string choices, structural list/tuple rows, and static-dict ``.items()``, ``.values()``, and ``.keys()`` views (including supported ``dict(iterable)`` - receivers); one layer of the element-preserving one-argument builtins - ``list``, ``tuple``, ``set``, ``frozenset``, ``sorted``, ``iter``, and - ``reversed`` resolves through that same iteration path; refused or partial + receivers); one layer of builtin wrappers classified by membership resolves + through that same iteration path: the strict one-argument forms of ``list``, + ``tuple``, ``set``, ``frozenset``, ``iter``, and ``reversed`` preserve + elements, ``sorted`` also accepts its order-only ``key`` and ``reverse`` + keywords, and ``filter(predicate, iterable)`` resolves as its unfiltered + operand because filtering can only remove elements; refused or partial binding over a fragment-bearing supported static container fails closed at the iteration site; and - contraband guarded-name literals anywhere statically visible in non-owner @@ -41,12 +44,12 @@ over-report combinations that no single row produces (a Cartesian over-catch). Over-reporting is the safe failure direction for a tripwire; a module that trips it restructures its table. Filtered identity -comprehensions over supported static sources likewise resolve as their -unfiltered row sets: predicates can only remove rows, so retaining every row -is the sound over-approximation. Partial dict views likewise retain every entry -with an opaque key because unknown runtime keys may be distinct; if they -collide, a value overwritten at runtime can remain in the guard's conservative -choice set. +comprehensions and ``filter`` calls over supported static sources likewise +resolve as their unfiltered row sets: predicates can only remove rows, so +retaining every row is the sound over-approximation. Partial dict views likewise +retain every entry with an opaque key because unknown runtime keys may be +distinct; if they collide, a value overwritten at runtime can remain in the +guard's conservative choice set. """ from __future__ import annotations @@ -519,9 +522,16 @@ def _static_string_shape( _OPAQUE_STATIC_VALUE = object() _OPAQUE_STRING_PART = "\N{OBJECT REPLACEMENT CHARACTER}" _OPAQUE_METHOD_ALIAS = ("", True) -_ELEMENT_PRESERVING_ONE_ARGUMENT_BUILTINS = frozenset( - {"frozenset", "iter", "list", "reversed", "set", "sorted", "tuple"} +_REFUSED_STATIC_ITERATION_VALUE = object() +_REFUSED_STATIC_ITERATION_WITH_GUARDED_FRAGMENTS = object() +_STRICT_ELEMENT_PRESERVING_ONE_ARGUMENT_BUILTINS = frozenset( + {"frozenset", "iter", "list", "reversed", "set", "tuple"} ) +_SORTED_ORDER_ONLY_KEYWORDS = frozenset({"key", "reverse"}) +_ITERATION_MEMBERSHIP_BUILTINS = _STRICT_ELEMENT_PRESERVING_ONE_ARGUMENT_BUILTINS | { + "filter", + "sorted", +} class _PartialStringChoices(tuple): @@ -1420,23 +1430,90 @@ def _value_bears_strings(value: object) -> bool: return False +def _static_iteration_wrapper_operand(node: ast.AST) -> ast.AST | None: + """Return a sound membership-superset operand for supported wrappers.""" + + if not isinstance(node, ast.Call) or not isinstance(node.func, ast.Name): + return None + name = node.func.id + if ( + name in _STRICT_ELEMENT_PRESERVING_ONE_ARGUMENT_BUILTINS + and len(node.args) == 1 + and not node.keywords + ): + return node.args[0] + if name == "sorted" and len(node.args) == 1: + keyword_names = tuple(keyword.arg for keyword in node.keywords) + if len(keyword_names) == len(set(keyword_names)) and all( + keyword_name in _SORTED_ORDER_ONLY_KEYWORDS + for keyword_name in keyword_names + ): + return node.args[0] + if name == "filter" and len(node.args) == 2 and not node.keywords: + # A predicate can remove members but cannot introduce them, so the + # unfiltered iterable is the sound candidate superset. + return node.args[1] + return None + + +def _iteration_value_carries_guarded_fragments(value: object) -> bool: + """Leaf-walk a resolved value without admitting refused wrappers.""" + + if value is _REFUSED_STATIC_ITERATION_WITH_GUARDED_FRAGMENTS: + return True + if value is _OPAQUE_STATIC_VALUE or value is _REFUSED_STATIC_ITERATION_VALUE: + return False + leaves: list[str] = [] + stack = [value] + while stack: + item = stack.pop() + if isinstance(item, str): + if len(item) >= 4: + leaves.append(item) + elif isinstance(item, (list, tuple, set, frozenset)): + stack.extend(item) + elif isinstance(item, dict): + stack.extend(item.keys()) + stack.extend(item.values()) + return any( + fragment in column for fragment in leaves for column in _OPERATOR_SOURCE_COLUMNS + ) + + def _static_iteration_value( node: ast.AST, constants: list[dict[str, object]], ) -> object: """Resolve the one structural value shared by binders and probes.""" + wrapper_operand = _static_iteration_wrapper_operand(node) + if wrapper_operand is not None: + # Classification depends on possible membership, never order or + # multiplicity. Element-preserving wrappers keep the operand's exact + # members; filter keeps a subset, for which the operand is a sound + # over-approximation. Preserve opaque sentinels through either path. + return _static_iteration_value(wrapper_operand, constants) if ( isinstance(node, ast.Call) and isinstance(node.func, ast.Name) - and node.func.id in _ELEMENT_PRESERVING_ONE_ARGUMENT_BUILTINS - and len(node.args) == 1 - and not node.keywords + and node.func.id in _ITERATION_MEMBERSHIP_BUILTINS ): - # Classification depends only on the iterated elements. These builtins - # materialize, reorder, or expose exactly those elements, so preserve - # the shared resolver's abstract value, including opaque sentinels. - return _static_iteration_value(node.args[0], constants) + # A recognized wrapper with a refused call shape must not enter the + # table. Retain only enough evidence for a later inline or bound + # iteration to fail closed when its operands carry guarded fragments. + argument_values = ( + *(_static_iteration_value(argument, constants) for argument in node.args), + *( + _static_iteration_value(keyword.value, constants) + for keyword in node.keywords + ), + ) + if any( + _iteration_value_carries_guarded_fragments(value) + for value in argument_values + ): + return _REFUSED_STATIC_ITERATION_WITH_GUARDED_FRAGMENTS + return _REFUSED_STATIC_ITERATION_VALUE structural_rows = _identity_structural_rows(node, constants) if structural_rows is not None and not isinstance(node, ast.DictComp): # Structural identity comprehensions map rows to themselves, so @@ -1516,23 +1593,8 @@ def _structure_carries_guarded_fragments( ) -> bool: """Leaf-walk the same resolved iterable structure used by the binder.""" - value = _static_iteration_value(node, constants) - if value is _OPAQUE_STATIC_VALUE: - return False - leaves: list[str] = [] - stack = [value] - while stack: - item = stack.pop() - if isinstance(item, str): - if len(item) >= 4: - leaves.append(item) - elif isinstance(item, (list, tuple, set, frozenset)): - stack.extend(item) - elif isinstance(item, dict): - stack.extend(item.keys()) - stack.extend(item.values()) - return any( - fragment in column for fragment in leaves for column in _OPERATOR_SOURCE_COLUMNS + return _iteration_value_carries_guarded_fragments( + _static_iteration_value(node, constants) ) @@ -3907,6 +3969,100 @@ def _finding_classifications(source: str) -> tuple[str, ...]: ) +@pytest.mark.parametrize( + "keyword", + ( + pytest.param("reverse=False", id="reverse-false"), + pytest.param("key=None", id="key-none"), + ), +) +@pytest.mark.parametrize("mode", ("inline", "bound")) +def test_round_21_exact_sorted_keyword_repros_classify( + keyword: str, + mode: str, +) -> None: + """The exact round-21 sorted calls resolve inline and after binding.""" + + lines = [ + 'DATA = {"person": "support_channel"}', + "def f(sink):", + ] + if mode == "bound": + lines.append(f" ROWS = sorted(DATA.items(), {keyword})") + iterable = "ROWS" + else: + iterable = f"sorted(DATA.items(), {keyword})" + lines.extend( + ( + f" for entity, suffix in {iterable}:", + ' sink(f"{entity}_{suffix}")', + ) + ) + + assert _finding_classifications("\n".join(lines)) == (_ROUND_20_NAMED_FINDING,) + + +@pytest.mark.parametrize( + "keyword", + ( + pytest.param("reverse=False", id="reverse-false"), + pytest.param("key=None", id="key-none"), + ), +) +@pytest.mark.parametrize("mode", ("inline", "bound")) +def test_round_21_partial_keys_under_sorted_keywords_dual_report( + keyword: str, + mode: str, +) -> None: + """A partial keys view retains both its known catch and its opacity.""" + + lines = [ + 'BASE = {"person": "age"}', + "def f(unknown_key, dynamic, sink):", + " DATA = {**BASE, unknown_key: dynamic}", + ] + if mode == "bound": + lines.append(f" ROWS = sorted(DATA.keys(), {keyword})") + iterable = "ROWS" + else: + iterable = f"sorted(DATA.keys(), {keyword})" + lines.extend( + ( + f" for entity in {iterable}:", + ' sink(f"{entity}_support_channel")', + ) + ) + + assert _finding_classifications("\n".join(lines)) == ( + _ROUND_20_NAMED_FINDING, + _ROUND_20_FAIL_CLOSED_FINDING, + ) + + +@pytest.mark.parametrize("mode", ("inline", "bound")) +def test_round_21_exact_filter_call_repro_classifies(mode: str) -> None: + """The filtered-comprehension finding also resolves in builtin form.""" + + lines = [ + 'DATA = {"person": "support_channel", "state": "fips"}', + "def f(sink):", + ] + expression = 'filter(lambda kv: kv[0] != "state", DATA.items())' + if mode == "bound": + lines.append(f" ROWS = {expression}") + iterable = "ROWS" + else: + iterable = expression + lines.extend( + ( + f" for entity, suffix in {iterable}:", + ' sink(f"{entity}_{suffix}")', + ) + ) + + assert _ROUND_20_NAMED_FINDING in _finding_classifications("\n".join(lines)) + + def test_round_20_exact_keys_wrapper_and_filtered_repros_classify(): """The round-20 reviewer constructions catch inline and after binding.""" From 36092ddbd2dc286d8672d1ca4a62344098863fe6 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 31 Jul 2026 09:42:31 -0400 Subject: [PATCH 94/94] Exercise the membership principle across wrapper matrices --- .../tests/test_us_spine_blindness.py | 318 ++++++++++++++++++ 1 file changed, 318 insertions(+) diff --git a/packages/populace-build/tests/test_us_spine_blindness.py b/packages/populace-build/tests/test_us_spine_blindness.py index 6b0faf77..3358cab7 100644 --- a/packages/populace-build/tests/test_us_spine_blindness.py +++ b/packages/populace-build/tests/test_us_spine_blindness.py @@ -4063,6 +4063,324 @@ def test_round_21_exact_filter_call_repro_classifies(mode: str) -> None: assert _ROUND_20_NAMED_FINDING in _finding_classifications("\n".join(lines)) +_ROUND_21_MEMBERSHIP_OPERANDS = ( + pytest.param("bare_dict", id="bare-dict"), + pytest.param("items", id="items-view"), + pytest.param("values", id="values-view"), + pytest.param("keys", id="keys-view"), + pytest.param("partial", id="partial-dict"), +) + + +def _round_21_membership_operand( + operand_case: str, + *, + hostile: bool, +) -> tuple[tuple[str, ...], str | None, str, str, str, bool]: + """Build one projection-sensitive operand for both wrapper matrices.""" + + if operand_case in {"bare_dict", "keys"}: + key = "person" if hostile else "state" + # A hostile benign-side value proves that only keys are projected. + value = "age" if hostile else "person" + return ( + (f'DATA = {{"{key}": "{value}"}}',), + None, + "DATA" if operand_case == "bare_dict" else "DATA.keys()", + "entity", + 'sink(f"{entity}_support_channel")', + False, + ) + if operand_case == "items": + key = "person" if hostile else "state" + value = "support_channel" if hostile else "fips" + return ( + (f'DATA = {{"{key}": "{value}"}}',), + None, + "DATA.items()", + "entity, suffix", + 'sink(f"{entity}_{suffix}")', + False, + ) + if operand_case == "values": + key = "known" if hostile else "person" + value = "person" if hostile else "state" + return ( + (f'DATA = {{"{key}": "{value}"}}',), + None, + "DATA.values()", + "entity", + 'sink(f"{entity}_support_channel")', + False, + ) + key = "person" if hostile else "state" + # The benign value again catches an accidental values projection. + value = "age" if hostile else "person" + return ( + (f'BASE = {{"{key}": "{value}"}}',), + "DATA = {**BASE, unknown_key: dynamic}", + "DATA", + "entity", + 'sink(f"{entity}_support_channel")', + True, + ) + + +def _round_21_membership_source( + call_template: str, + *, + operand_case: str, + hostile: bool, + mode: str, +) -> tuple[str, bool]: + setup, data_binding, operand, target, sink_statement, partial = ( + _round_21_membership_operand(operand_case, hostile=hostile) + ) + lines = [ + *setup, + "def order_key(value):", + " return repr(value)", + "def reject(value):", + " return False", + "def f(unknown_key, dynamic, sink, dynamic_key, dynamic_reverse):", + ] + if data_binding is not None: + lines.append(f" {data_binding}") + expression = call_template.format(operand=operand) + if mode == "bound": + lines.append(f" WRAPPED = {expression}") + iterable = "WRAPPED" + else: + iterable = expression + lines.extend( + ( + f" for {target} in {iterable}:", + f" {sink_statement}", + ) + ) + return "\n".join(lines), partial + + +def _round_21_membership_expected( + *, + hostile: bool, + partial: bool, +) -> tuple[str, ...]: + if hostile and partial: + return (_ROUND_20_NAMED_FINDING, _ROUND_20_FAIL_CLOSED_FINDING) + if hostile: + return (_ROUND_20_NAMED_FINDING,) + return () + + +@pytest.mark.parametrize( + "keyword", + ( + pytest.param("key=None", id="key-none"), + pytest.param("key=lambda value: repr(value)", id="key-lambda"), + pytest.param("key=order_key", id="key-named-function"), + pytest.param("reverse=False", id="reverse-false"), + pytest.param("reverse=True", id="reverse-true"), + pytest.param( + "key=dynamic_key, reverse=dynamic_reverse", + id="key-and-reverse", + ), + ), +) +@pytest.mark.parametrize("operand_case", _ROUND_21_MEMBERSHIP_OPERANDS) +@pytest.mark.parametrize( + "hostile", + (pytest.param(True, id="hostile"), pytest.param(False, id="benign")), +) +@pytest.mark.parametrize("mode", ("inline", "bound")) +def test_round_21_sorted_keyword_membership_matrix( + keyword: str, + operand_case: str, + hostile: bool, + mode: str, +) -> None: + """Order-only sorted keywords never change candidate membership.""" + + source, partial = _round_21_membership_source( + f"sorted({{operand}}, {keyword})", + operand_case=operand_case, + hostile=hostile, + mode=mode, + ) + assert _finding_classifications(source) == _round_21_membership_expected( + hostile=hostile, + partial=partial, + ), source + + +@pytest.mark.parametrize( + "predicate", + ( + pytest.param("None", id="none"), + pytest.param("lambda value: False", id="lambda"), + pytest.param("reject", id="named-function"), + ), +) +@pytest.mark.parametrize("operand_case", _ROUND_21_MEMBERSHIP_OPERANDS) +@pytest.mark.parametrize( + "hostile", + (pytest.param(True, id="hostile"), pytest.param(False, id="benign")), +) +@pytest.mark.parametrize("mode", ("inline", "bound")) +def test_round_21_filter_membership_matrix( + predicate: str, + operand_case: str, + hostile: bool, + mode: str, +) -> None: + """Every predicate resolves as the unfiltered candidate superset.""" + + source, partial = _round_21_membership_source( + f"filter({predicate}, {{operand}})", + operand_case=operand_case, + hostile=hostile, + mode=mode, + ) + assert _finding_classifications(source) == _round_21_membership_expected( + hostile=hostile, + partial=partial, + ), source + + +@pytest.mark.parametrize( + "call_template", + ( + pytest.param("sorted({operand}, reverse=False)", id="sorted"), + pytest.param("filter(None, {operand})", id="filter"), + ), +) +@pytest.mark.parametrize("view", ("keys", "values", "items")) +@pytest.mark.parametrize("mode", ("inline", "bound")) +def test_round_21_partial_views_under_membership_wrappers_dual_report( + call_template: str, + view: str, + mode: str, +) -> None: + """Both membership doctrines preserve every partial-view candidate.""" + + if view == "keys": + base = 'BASE = {"person": "age"}' + target = "entity" + sink_statement = 'sink(f"{entity}_support_channel")' + elif view == "values": + base = 'BASE = {"known": "person"}' + target = "entity" + sink_statement = 'sink(f"{entity}_support_channel")' + else: + base = 'BASE = {"person": "support_channel"}' + target = "entity, suffix" + sink_statement = 'sink(f"{entity}_{suffix}")' + lines = [ + base, + "def f(unknown_key, dynamic, sink):", + " DATA = {**BASE, unknown_key: dynamic}", + ] + expression = call_template.format(operand=f"DATA.{view}()") + if mode == "bound": + lines.append(f" WRAPPED = {expression}") + iterable = "WRAPPED" + else: + iterable = expression + lines.extend( + ( + f" for {target} in {iterable}:", + f" {sink_statement}", + ) + ) + + assert _finding_classifications("\n".join(lines)) == ( + _ROUND_20_NAMED_FINDING, + _ROUND_20_FAIL_CLOSED_FINDING, + ) + + +@pytest.mark.parametrize("mode", ("inline", "bound")) +def test_round_21_filter_retains_duplicate_removed_row_candidates(mode: str) -> None: + """A filtered-out hostile duplicate remains in the safe candidate set.""" + + data = ( + ("person", "support_channel"), + ("person", "age"), + ("state", "fips"), + ) + assert tuple(filter(lambda row: row[1] == "age", data)) == (("person", "age"),) + + lines = [ + 'DATA = (("person", "support_channel"), ("person", "age"), ("state", "fips"))', + "def f(sink):", + ] + expression = 'filter(lambda row: row[1] == "age", DATA)' + if mode == "bound": + lines.append(f" ROWS = {expression}") + iterable = "ROWS" + else: + iterable = expression + lines.extend( + ( + f" for entity, suffix in {iterable}:", + ' sink(f"{entity}_{suffix}")', + ) + ) + + assert _ROUND_20_NAMED_FINDING in _finding_classifications("\n".join(lines)) + + +_ROUND_21_REFUSED_WRAPPER_CALLS = tuple( + pytest.param(wrapper, shape, id=f"{wrapper}-{shape}") + for wrapper in ( + *_ROUND_20_ELEMENT_PRESERVING_BUILTINS, + "filter", + ) + for shape in ("keyword", "extra-positional") +) + + +@pytest.mark.parametrize(("wrapper", "shape"), _ROUND_21_REFUSED_WRAPPER_CALLS) +@pytest.mark.parametrize("mode", ("inline", "bound")) +def test_round_21_refused_wrapper_shapes_stay_fail_closed( + wrapper: str, + shape: str, + mode: str, +) -> None: + """Rejected call shapes preserve fragments without entering the table.""" + + operand = "DATA.items()" + if wrapper == "filter": + expression = ( + f"filter(None, iterable={operand})" + if shape == "keyword" + else f"filter(None, {operand}, None)" + ) + elif shape == "keyword": + expression = f"{wrapper}({operand}, unexpected=None)" + else: + expression = f"{wrapper}({operand}, None)" + lines = [ + 'DATA = {"person": "support_channel"}', + "def f(sink):", + ] + if mode == "bound": + lines.append(f" WRAPPED = {expression}") + iterable = "WRAPPED" + else: + iterable = expression + lines.extend( + ( + f" for entity, suffix in {iterable}:", + ' sink(f"{entity}_{suffix}")', + ) + ) + + classifications = _finding_classifications("\n".join(lines)) + assert classifications == (_ROUND_20_FAIL_CLOSED_FINDING,) + assert _ROUND_20_NAMED_FINDING not in classifications + + def test_round_20_exact_keys_wrapper_and_filtered_repros_classify(): """The round-20 reviewer constructions catch inline and after binding."""