One-pool multispine builder: assemble-first build path with the agreement gate terminal (#578 increment 2)

changelog.d/578-multispine-pool-build.added.md

@@ -0,0 +1,10 @@
+Add a SHA-pinned, pre-calibration US multispine pool builder that consumes a
+dedicated operator-untouched ASEC raw-stage artifact, assembles ASEC and ACS
+before PUF cloning and the full source-blind operator chain, publishes nullable
+input-only artifacts through a run-bound invalidate-then-publish protocol, and
+runs the fixed spine-agreement gate over the complete pool transfer charter,
+including categorical joint checks, as its terminal stage; the readiness
+loader verifies the manifest, H5, and diagnostics as one publication, the
+deprecated ACS local-release builder remains executable until increment 4,
+and this code increment has synthetic fixture coverage but does not certify a
+full-data build.

docs/us-multispine-operator-ordering.md

@@ -1,16 +1,16 @@
 # US multispine operator ordering
 
-This note records the build ordering visible in
-`tools/build_us_puf_support_base.py`,
-`tools/build_us_acs_multispine_base.py`, and their directly called US runtime
-modules. It defines the seam introduced by populace#395; it does not certify
-the distributions of an output artifact.
+This note records the executable pool ordering in
+`tools/build_us_multispine_pool.py` and the serial lineage it replaces. The
+assembly and agreement contracts originated in populace#581; populace#578
+increment 2 wires them into a build path. This documentation and its fixture
+tests do not certify a full-data output artifact.
 
-## Current ordering
+## Retired serial ordering
 
-The current lineage is two serial builds. The first build produces an
-operated ASEC-by-PUF-detail donor. The second build creates ACS records,
-transfers inputs from that donor, and only then appends ACS.
+The earlier lineage used two serial builds. The first build produced an
+operated ASEC-by-PUF-detail donor. The second build created ACS records,
+transferred inputs from that donor, and only then appended ACS.
 
 ### `build_us_puf_support_base.py`
 
@@ -35,8 +35,8 @@ stages.
 
 ### `build_us_acs_multispine_base.py`
 
-The ACS builder takes that exported H5 as `--base-h5`. Its runtime call graph
-is:
+The former ACS builder took that exported H5 as `--base-h5`. Its runtime call
+graph was:
 
 1. Load and validate the dense ASEC-by-PUF-detail base and declared transfer
    coverage.
@@ -55,14 +55,114 @@ is:
 Calibration is downstream of this tool. Simulation is not run by either
 builder in this call graph.
 
-The important ordering fact is structural: ACS receives model-input
+The important ordering flaw is structural: ACS receives model-input
 transfers from a donor after the donor has crossed the ASEC-only operator
 sequence. Appending the transferred ACS records later does not cause those
 operators to run over the combined population.
 
-## Current provenance axes
-
-The current lineage uses two related metadata schemes:
+`build_us_acs_multispine_base.py` remains a deprecated but executable
+compatibility path until populace#578 increment 4 retires the ACS local-release
+overlay. The public command warns and delegates to the preserved implementation
+under `tools/_legacy`; its summary and reviewed-null receipts remain the inputs
+expected by `build_us_acs_local_release.py`. New multispine work uses the pool
+builder below, but the supported legacy release recipe is not left half-working.
+
+## Executable increment-2 pool build
+
+`build_us_multispine_pool.py` consumes only explicit local files and their
+declared SHA-256 values:
+
+- the dedicated `populace_us_asec_raw_stage` artifact emitted alongside the
+  producer's `source_construction` checkpoint. Its stage tag is
+  `raw_source_mapping` and its operator status is `operator_untouched`;
+- the ACS household and person PUMS archives, whose caller-supplied hashes
+  must also match the checked-in ACS source manifest;
+- the canonical ACS 2022 rent donor used by the post-assembly housing
+  operator; and
+- the processed PUF H5 and source-year PUF CSV used by the existing donor
+  loader.
+
+The raw artifact is a second producer output, not a relabeling of
+`pre_clone_enrichment`. It contains pooled ASEC unit structure and measured raw
+columns. The only enrichment allowed there is faithful source mapping:
+`LKWEEKS` and `ED_VAL` are restored by exact, pinned Census identity joins.
+No `weeks_unemployed`, `educational_assistance`, carried-income split,
+eligibility, pregnancy, take-up, childcare, retirement, or immigration output
+is present. The producer still emits its historical enriched checkpoint and
+final H5 for the sparse/dense single-spine release lineage; their operator
+sequence and bytes are unchanged.
+
+The dedicated raw artifact is produced by the checkpointed producer recipe
+(`--stage all --checkpoint-dir ...`) at
+`<checkpoint-dir>/asec_raw_stage.checkpoint.h5`. The legacy monolithic recipe
+continues to produce only its historical release outputs.
+
+The tool does not download any source. It verifies all file pins and validates
+the raw artifact kind, stage, frame identity, operator status, and complete
+operator-output absence before loading the peer frames. Measured ACS mappings
+are allowed only when named by the ACS native-input receipt. It then runs this
+fixed sequence:
+
+1. `assemble_spines({"asec": ..., "acs": ...})` creates the first shared
+   population state and binds the immutable assembly receipt.
+2. `clone_us_frame_for_puf_support(...)` applies the PUF-detail clone to the
+   whole assembled pool. Clone-index provenance, not source-spine identity,
+   controls later PUF-detail routing.
+3. CPS-carried predictor inputs are derived after assembly on rows with the
+   required measured CPS source fields. The pool wrapper validates the complete
+   assembly receipt first, then gives the historical CPS kernel an ephemeral
+   `PERIDNUM`-available structural projection. That projection deliberately
+   carries neither the full-pool assembly receipt nor its mass history. Only
+   the kernel's declared output family is merged back into the still-receipted
+   full pool. Unavailable peer rows remain nullable; no operator uses
+   source-channel identity to choose behavior.
+4. The primary PUF QRF chain and capital-gains tail transfer run over the
+   combined frame. The remaining historical ASEC input families then run in
+   their declared order on source-evidenced rows: prior-year income,
+   relationship and Medicare inputs, housing, eligibility, pregnancy, WIC,
+   housing-assistance support transfer, child support, disability benefits,
+   workers compensation, weeks unemployed, childcare, adult care, energy
+   subsidy, retirement contributions/distributions, immigration, and
+   education inputs.
+5. The pool-specific ACS transfer plan fills only still-null peer cells from
+   those post-assembly results. Existing measured/native cells remain
+   byte-for-byte unchanged, and transfer receipts record fitted and imputed
+   rows.
+6. Schedule-D and QBI deterministic reconciliation run over that same pool.
+7. The seed stage preserves existing take-up values, applies the sourced
+   TANF and EITC mechanisms, and explicitly receipts live engine defaults
+   used for unresolved, non-transfer-owned take-up inputs. Those defaults
+   are not described as fitted or administrative mechanisms.
+8. SSI is materialized only on an ephemeral agreement view in fixed
+   household batches. Any engine defaults required solely for that
+   calculation are separately receipted; formula output is not written into
+   the input pool.
+9. The spine-agreement gate is terminal. Its immutable pool registry is built
+   from the complete pool-specific transfer plan plus derived, take-up, and SSI
+   surfaces. Numeric columns retain the fixed incidence and conditional-
+   quantile tolerances. Categorical columns use a fixed weighted total-
+   variation-distance ceiling of `0.25`; the immigration fields are also
+   checked jointly so matching marginals cannot conceal incompatible pairs.
+   The gate batches all failures and controls the manifest's simulation-ready
+   status.
+
+The output H5 is a nullable, input-only, pre-calibration pool. Its companion
+manifest carries input pins, the assembly receipt, per-source and per-clone
+counts, operator receipts, and the complete agreement result. Publication
+first atomically replaces any prior manifest with a non-ready tombstone, then
+stages the H5 and diagnostics under one publication run ID and renames them,
+and finally writes the readiness manifest. The manifest records the H5 and
+diagnostics run IDs and SHA-256 digests. The readiness loader requires a green
+manifest whose run ID and digests match the H5 metadata and diagnostics
+payload. An interrupted, substituted, or failed publication therefore
+self-reports not ready even beside stale files. A failed agreement gate writes
+diagnostics and a non-ready final manifest and exits nonzero. Calibration is
+deliberately absent; the downstream k-ladder
+may consume only a pool whose terminal agreement result passed.
+
+## Provenance axes
+
+The retired lineage used two related metadata schemes:
 
 - PUF support cloning adds, on every entity,
   `*_source_id`, `*_support_channel`, and
@@ -77,13 +177,13 @@ The current lineage uses two related metadata schemes:
   including the target family, donor spine/channel, predictors, seeds,
   weight kind, recipient patterns, and unmodeled-row count.
 
-Those fields currently mix two concepts: the population source that carried
-a record and the PUF-detail copy created by an operator. The new seam keeps
-those concepts separate.
+Those fields mixed two concepts: the population source that carried a record
+and the PUF-detail copy created by an operator. The assembly seam keeps those
+concepts separate.
 
-## Canonical target ordering
+## Canonical executable ordering
 
-The target US multispine build order is:
+The US multispine build order is:
 
 ```text
 source ingestion and faithful schema harmonization
@@ -94,15 +194,22 @@ source ingestion and faithful schema harmonization
     -> seed take-up and other stochastic inputs
     -> simulate
     -> spine-agreement gate
-    -> calibrate
+    -> emit input-only pool and receipts
 ```
 
+Calibration is a downstream consumer boundary, not a stage in this tool.
+
 `assemble_spines(...)` is the boundary between source preparation and
 population operators. It receives nullable, schema-compatible peer frames
 and produces one combined frame before cloning, fitted transfer, derivation,
-seeded assignment, simulation, or calibration. Downstream operator
+seeded assignment, simulation, or calibration. Downstream pool-stage
 entrypoints receive that combined frame and operate on measured
-characteristics without selecting behavior by source spine.
+characteristics without selecting behavior by source spine. A historical
+source kernel that requires CPS-only raw fields receives an ephemeral
+availability projection after the combined-frame boundary is validated. Such
+a projection is not published or described as a full-pool lineage state; its
+declared outputs are merged back into the combined frame, whose immutable
+assembly receipt remains the authority.
 
 ASEC and ACS are peer household spines. A future household source can join
 the same assembly contract. PUF tax detail is not a peer spine: it remains a
@@ -149,8 +256,11 @@ marital-unit, and benefit-unit naming variants. It recognizes direct,
 aliased-helper, dynamic-subscript, `getattr`, and `*_spine_source_id` reads.
 Every US runtime module is classified as a reviewed population operator or an
 explicit non-operator/provenance owner, so a new unclassified module fails the
-guard. `transfer_acs_inputs` selects fit donors by the centrally derived clone
-role; assembled source-channel names never determine donor eligibility.
+guard. A second fail-closed scan starts at `build_us_multispine_pool.py` and
+covers the tool plus its transitive US-runtime import graph under the same
+owner registry. `transfer_acs_inputs` selects fit donors by the centrally
+derived clone role; assembled source-channel names never determine donor
+eligibility.
 
 Assembly accepts only integer-typed, nonnegative structural source IDs and
 names the source spine and offending IDs on failure. PUF cloning revalidates
@@ -217,15 +327,10 @@ This ordering makes the gate diagnostic of the shared operator surface:
 calibration cannot hide a disagreement, and no per-spine target, loss term,
 seed, or tolerance may be introduced to shape a passing result.
 
-## Increment-1 compatibility boundary
-
-This increment adds an opt-in assembly seam, operator contracts, structural
-enforcement, and the agreement-gate specification. It does not rewire
-`build_us_puf_support_base.py`,
-`build_us_acs_multispine_base.py`, or current sparse/dense release tools to
-the new sequence. Their current call paths and artifact behavior remain the
-compatibility lineage until a later increment explicitly adopts the seam.
+## Increment-2 validation boundary
 
-The seam is the foundation for the broader populace#578 build shape,
-including one suite per country and the US full-geography/exact-k work. Those
-release changes are outside this increment.
+This increment makes the assembly seam executable and covers it with small,
+synthetic two-spine fixtures. It does not execute or certify a full-data
+build, download a source dataset, calibrate the pool, select k, or change the
+current sparse/dense release artifacts. Those data-scale and release steps
+remain downstream of this code increment.

packages/populace-build/src/populace/build/us_runtime/__init__.py

@@ -59,6 +59,15 @@
     us_alimony_stage_spec,
     us_alimony_summary,
 )
+from populace.build.us_runtime.asec_checkpoint import (
+    ASEC_RAW_STAGE_ARTIFACT_KIND,
+    ASEC_RAW_STAGE_CHECKPOINT_FILENAME,
+    ASEC_RAW_STAGE_OPERATOR_STATUS,
+    ASEC_RAW_STAGE_SCHEMA_VERSION,
+    ASEC_RAW_STAGE_STAGE,
+    load_asec_pre_clone_checkpoint,
+    load_asec_raw_stage_checkpoint,
+)
 from populace.build.us_runtime.asec_pool import (
     AsecSource,
     build_pooled_asec_unit_frame,
@@ -409,6 +418,10 @@
     nonzero_share,
     us_nonzero_shares,
 )
+from populace.build.us_runtime.operator_boundary import (
+    PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES,
+    assert_operator_free_source_frame,
+)
 from populace.build.us_runtime.org_wages import (
     BLS_STATE_UNION_REPRESENTATION_RATE_2024,
     FLSA_EXECUTIVE_ADMINISTRATIVE_PROFESSIONAL_OCCUPATION_CODES,
@@ -516,6 +529,7 @@
     validate_puf_capital_gains_tail_manifest,
     write_puf_capital_gains_tail_manifest,
 )
+from populace.build.us_runtime.puf_donor_io import load_puf_tax_unit_donor
 from populace.build.us_runtime.puf_e01000_reconciliation import (
     PUF_E01000_RECONCILIATION_SCHEMA_VERSION,
     build_puf_e01000_reconciliation_basis,
@@ -857,6 +871,7 @@
     us_source_operation_handlers,
 )
 from populace.build.us_runtime.spine_agreement import (
+    DEFAULT_CATEGORICAL_TOTAL_VARIATION_TOLERANCE,
     DEFAULT_INCIDENCE_RATIO_BOUNDS,
     DEFAULT_QUANTILE_ENVELOPE_TOLERANCE,
     DEFAULT_SPINE_AGREEMENT_QUANTILES,
@@ -1041,6 +1056,11 @@
 from populace.frame import Frame
 
 __all__ = [
+    "ASEC_RAW_STAGE_ARTIFACT_KIND",
+    "ASEC_RAW_STAGE_CHECKPOINT_FILENAME",
+    "ASEC_RAW_STAGE_OPERATOR_STATUS",
+    "ASEC_RAW_STAGE_SCHEMA_VERSION",
+    "ASEC_RAW_STAGE_STAGE",
     "BuildConfig",
     "AsecSource",
     "BASE_ASEC_SUPPORT_CHANNEL",
@@ -1774,12 +1794,14 @@
     "PUF_SOURCE_YEAR",
     "PUF_SOURCE_YEAR_AGI_REQUIRED_COLUMNS",
     "PUF_SYNTHETIC_RECID_START",
+    "PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES",
     "US_PUF_SUPPORT_FIT_NAME",
     "US_PUF_SUPPORT_STAGE_NAME",
     "US_STATE_INCOME_TAX_TARGET_SPECS",
     "US_STATE_INCOME_TAX_TARGET_REFERENCES",
     "compile_us_fiscal_target_registry",
     "assign_congressional_districts_to_households",
+    "assert_operator_free_source_frame",
     "build_pooled_asec_unit_frame",
     "clone_us_frame_for_puf_support",
     "congressional_district_assignment_summary",
@@ -1795,6 +1817,9 @@
     "load_default_reform_specs",
     "load_congressional_district_vintage_crosswalk",
     "load_default_congressional_district_vintage_crosswalk",
+    "load_asec_pre_clone_checkpoint",
+    "load_asec_raw_stage_checkpoint",
+    "load_puf_tax_unit_donor",
     "normalize_district_code",
     "parse_baf_cd_layer",
     "parse_national_cd_bef_districts",
@@ -1861,6 +1886,7 @@
     "us_register_consistency_gate",
     "us_register_contradictions",
     "write_us_source_coverage_diagnostics",
+    "DEFAULT_CATEGORICAL_TOTAL_VARIATION_TOLERANCE",
     "DEFAULT_INCIDENCE_RATIO_BOUNDS",
     "DEFAULT_QUANTILE_ENVELOPE_TOLERANCE",
     "DEFAULT_SPINE_AGREEMENT_QUANTILES",

packages/populace-build/src/populace/build/us_runtime/acs_pums.py

@@ -7,6 +7,12 @@
 ACS input-mapping stage applies the Census adjustment factors and records
 which PolicyEngine inputs are native versus transferred.
 
+Raw ``SERIALNO`` remains on the household table.  Person lineage uses the
+stable, sorted household ID assigned from that key, string-valued ``SPORDER``,
+and a sequential row ID.  Those generated fields match the ASEC source-frame
+lineage dtypes required by pre-operator spine assembly without rewriting a
+measured Census value.
+
 Full national archives contain multiple CSV members.  Each member is read in
 bounded chunks, but the final selected-column tables necessarily materialize:
 the returned :class:`~populace.frame.Frame` itself is the dense base-pool
@@ -264,15 +270,12 @@ def build_acs_pums_unit_frame(
     person["household_id"] = person["household_id"].astype("int64")
     person = _with_structural_columns(person)
     person["source_year"] = source.vintage
-    person["source_household_id"] = person["SERIALNO"].astype(str)
-    person["source_person_id"] = _required_integer(person, "SPORDER")
-    person["source_row_id"] = (
-        ACS_2024_1YR_SPINE
-        + ":"
-        + person["SERIALNO"].astype(str)
-        + ":"
-        + person["SPORDER"].astype("int64").astype(str)
-    )
+    person["source_household_id"] = person["household_id"].to_numpy(dtype=np.int64)
+    person["source_person_id"] = pd.Series(
+        _required_integer(person, "SPORDER"),
+        index=person.index,
+    ).astype(str)
+    person["source_row_id"] = np.arange(len(person), dtype=np.int64)
 
     household_weights = _household_weights(household, person)
     # SERIALNO belongs on the household table in the returned Frame. Its