diff --git a/changelog.d/652-capital-gains-tail-thin-strata.fixed.md b/changelog.d/652-capital-gains-tail-thin-strata.fixed.md new file mode 100644 index 00000000..a2aa0ab3 --- /dev/null +++ b/changelog.d/652-capital-gains-tail-thin-strata.fixed.md @@ -0,0 +1 @@ +Declare capital-gains tail support per filing status: require one eligible unique single-tax-unit PUF-detail recipient household per selected q99.5 donor, skip a thin status as a whole with a named and counted `insufficient_support` receipt, and reject altered receipts at terminal gates. Preserve byte-exact attachment for adequate statuses and unchanged full-scale output. Replace the fixed post-clone source-then-transfer sequence with an import-validated 38-producer/71-edge dependency DAG whose readiness fence derives the byte-stable order, separates missing inputs from invalid finite numerics, rejects named cycles and unfilled inputs, transfers PUF SSTB income before adult care and tuition before education, and makes the ACS earnings-universe producer, all sixteen source producers, and their once-only finalizer explicit. Content-hash every declared input alternative, reconciled readiness row, output, callback receipt, execution row, and entry/output frame; anchor the signed chain in immutable frame metadata and carry its independent authority through version-9 stacked authority, outer stacked materializer v10, version-6 stacked pool checkpoints, and schema-8 pool manifests, H5 materializer v2, and consumers while preserving the retiring legacy schema-4/materializer-3 bytes and rejecting stacked-envelope downgrade attempts. Bind the primary donor bytes, routed QRF bank and sidecar, exact optional allocation/passthrough reads, worker controls, once-resolved tail spec/SOI assets and gates, all nineteen transfer predictor/codec/model configurations and bank identities, all sixteen source runtime configurations—including packaged `SourceStageSpec` identities for the fifteen manifest-backed callbacks—finalizer doctrine, and the ACS universe runtime rule/config through exact kind-specific schema-v1/v2/v3 resource evidence in late-registry schema v14/receipt schema v3. Bind every virtual-resource resolution mode into the outer checkpoint identity, prove current checkpoint discovery succeeds and stale resource semantics fail, require adult-care transfer's tax-unit role before dispatch, keep Schedule-D ownership in the whole-pool derive rather than bounded transfer side effects, and reject shallow, forged, stale, identityless, runtime-drifted, or cross-producer absence resources before their callbacks. Scope ACS-only `TYPEHUGQ` readiness and structural output coverage to ACS-origin rows, keep the 1,688 ASEC-origin structural nulls untouched, reject any ACS-side null, and retire whole-pool `RELSHIPP`, `TEN`, and `H_TENURE` transfer fallbacks in favor of canonical dual-origin predictors. Preserve all canonical late-stage boolean outputs as nullable booleans at primary-PUF and source-merge materialization seams, fail closed on numeric or object family drift across every registry-declared late write, and retain boolean dtype through per-stratum tail cloning. Serialize pandas nullable booleans through conditional frame-checkpoint schema v3 as canonical NumPy-bool values plus an explicit lossless null mask only when absences exist, restore the logical nullable family on load, reject malformed or downgraded encodings, and retain byte-identical schema-v2 generic, UK, and legacy artifacts when no nullable boolean is present. Declare the exact final owner for every origin and clone role of qualified tuition, traditional IRA contributions, and self-employed pension contributions in a content-addressed 18-row receipt bound into late-registry schema v15 and the tail manifest; prove education is byte-exact consume-only, mirror the two source-owned ASEC retirement values from clone 1 to clone 2 by assembly-unique source ID, and reject any unregistered recipient-owned or tail-owned dual-write intersection without weakening the terminal preservation guard. Materialize pandas nullable booleans at every one of the eight registry-declared Frame-table serializers: emit byte-identical native NumPy bool values for complete columns and preserve genuine absences explicitly, using a version-1 table-local codec receipt with object-backed bool plus `pd.NA` at PyTables fixed-format boundaries—even for all-missing columns—and canonical bool values plus uint8 masks in h5py codecs. Advance fiscal target-frame checkpoints to schema 2/materializer 11, bind stacked terminal publication schema 8 to H5 materializer 2 in both manifest and H5 metadata, reject malformed or cross-envelope version evidence, retain frozen HDF keys/artifact kinds/`entity_hdf_format`, and keep legacy schema-4 publication bytes and unaffected UK payloads unchanged. diff --git a/changelog.d/652-stacked-qbi-input-provenance.fixed.md b/changelog.d/652-stacked-qbi-input-provenance.fixed.md new file mode 100644 index 00000000..70f26862 --- /dev/null +++ b/changelog.d/652-stacked-qbi-input-provenance.fixed.md @@ -0,0 +1 @@ +Materialize the certified exact-zero S-corporation income universe across the stacked US pool before whole-pool QBI reconciliation, and bind an exhaustive derive/seed/simulate input manifest into stacked checkpoint identity so missing late-stage inputs fail before long builds. diff --git a/docs/us-multispine-operator-ordering.md b/docs/us-multispine-operator-ordering.md index 645328c4..dce3e766 100644 --- a/docs/us-multispine-operator-ordering.md +++ b/docs/us-multispine-operator-ordering.md @@ -106,6 +106,23 @@ are allowed only when named by the ACS native-input receipt. ### Default sequence +The exact outer `US_STACKED_POOL_OPERATOR_ORDER` is byte-stable and contains +these ten entries. The numbered paragraphs below explain those phases; nested +callbacks are not additional outer entries. + +```text +assemble_stacked_spine +prepare_multispine_source_inputs_for_clone +gap_fill_stacked_spine +run_stacked_late_producer_dag +prepare_stacked_tail_derivation +derive_multispine_pool_inputs +seed_multispine_pool_inputs +materialize_multispine_agreement_outputs +stacked_completeness_gate +by_origin_battery +``` + 1. `assemble_stacked_spine(...)` selects whole households independently from both survey arms with the single `sample_fraction` and `sample_seed`, restores each sample to its full-source design-weight mass, and assembles @@ -122,7 +139,7 @@ are allowed only when named by the ACS native-input receipt. particular, the pinned ACS rent artifact trains `with_us_housing_inputs`, which materializes `pre_subsidy_rent` on ASEC; native ACS `RNTP`/`GRNTP` remain predictors and are not relabeled as that model input. - Targets produced only by the later PUF pass or source-completion chain are + Targets produced only by the later PUF pass or late source producers are excluded from this early authority surface. No population operator selects behavior from the source-channel labels. 3. `gap_fill_stacked_spine(...)` runs the two immutable directions over the @@ -142,9 +159,11 @@ are allowed only when named by the ACS native-input receipt. finish with zero `unmodeled_rows` and zero residual nulls: transfer accounting alone is not terminal absence authority. The #608 per-target banks sit beneath the stack-bound checkpoint identity. -4. `run_stacked_puf_pass(...)` attaches the separately controlled PUF clone - arm (`clone_attachment_fraction`, default `1.0`) and runs one primary QRF - pass across both survey origins. The strict recipient surface applies the +4. The derived second node of `run_stacked_late_producer_dag(...)` invokes + `run_stacked_puf_pass(...)`; it is not a second outer operator-order entry. + The callback attaches the separately controlled PUF clone arm + (`clone_attachment_fraction`, default `1.0`) and runs one primary QRF pass + across both survey origins. The strict recipient surface applies the [2024 ACS PUMS Data Dictionary](https://www2.census.gov/programs-surveys/acs/tech_docs/pums/data_dict/PUMS_Data_Dictionary_2024.pdf) universes for `WAGP` and `SEMP`. The ASEC evidence is the established producer, not a new convention: [`derive_us_cps_carried_inputs`](../packages/microcosm-build/src/microcosm/build/us_runtime/cps_carried.py#L150-L155) @@ -186,29 +205,103 @@ are allowed only when named by the ACS native-input receipt. receipt. PUF donors stay full. The clone-2 capital-gains-tail operator runs inside this pass; exact tail-owned and QRF-owned cells are checked after source completion and every later phase. - This semantic change is authority-gated: the primary-QRF root and target - checkpoint schema, outer stacked checkpoint materializer, and canonical - stacked authority are all version 6. The outer base identity binds the - primary-QRF schema plus the ACS universe and QBI reconciliation contract - identities. Every v1--v5 payload is stale and refused, including the former - strict v5 two-control payload. -5. The post-clone source-completion chain runs, then the declared post-PUF - transfer fills the targets first materialized by that chain or the PUF pass. - Its complete model donor is the ASEC-origin PUF-detail role. Authority is + The tail declares support separately for each filing status. Its required + minimum is the number of selected q99.5 PUF tail donors in that status. Its + observed support is the number of unique, single-tax-unit PUF-detail + recipient households in that status whose half-weight can carry the global + maximum assigned tail-donor weight. When observed support falls below the + minimum, the operator skips the whole status and emits a named, counted + `insufficient_support` receipt with the status, observed count, and required + minimum. It never borrows recipients from another status or partially + attaches the status. All 22 AGI bands remain nearest-first fallback choices + within a filing status; they are not separate hard partitions. A status + with zero selected donors, such as `SURVIVING_SPOUSE` in the pinned tail, + receipts `not_applicable` rather than a skip. + + At the standard 1% rung, `SINGLE` and `HEAD_OF_HOUSEHOLD` attach, + `JOINT` and `SEPARATE` receipt `insufficient_support`, and + `SURVIVING_SPOUSE` receipts `not_applicable`. Every positive-requirement + status meets support at the 10% and full rungs. Filtering occurs only after + the operator constructs the original global-capacity candidate pool, so + attached statuses retain the pre-change assignment bytes. Full-scale output + therefore remains unchanged. + + The authority versions distinguish the two contracts. The primary-QRF root + and target checkpoint schema remains version 6. The capital-gains tail + manifest uses schema version 2 and binds its support contract and receipt. + The canonical stacked authority remains version 9, the outer stacked + checkpoint materializer uses version 10, and the stacked pool stage + checkpoint materializer uses version 5. + The outer base identity binds primary-QRF version 6, the ACS universe and + QBI reconciliation contracts, the tail schema and support contract, and + late-producer registry schema version 14, including the signed static and + derivation-mode semantics of every virtual DAG resource. The companion pool + manifest uses schema version 7. + Older outer authority or materializer payloads are stale; primary-QRF + version 6 remains current. + + The saved 10% failure checkpoint makes the ordering mechanism concrete. + Before PUF it contains 385,992 clone-0 people: 342,732 ACS-origin and 43,260 + ASEC-origin. After clone attachment, the PUF-detail clone-1 cells for + `sstb_self_employment_income_before_lsr` are finite while the corresponding + clone-0 recipient cells remain null. The adult-care source projection is + ASEC-scoped, so its strict earned-income accumulator encounters exactly the + 43,260 ASEC-origin clone-0 nulls and raises. It does not encounter 43,260 + ACS-origin rows; that origin description is contradicted by the checkpoint. + The former driver called all post-clone source completion before the late + transfer. The declared batch-5-to-adult-care edge below therefore derives + the repair from data dependency rather than installing another manual + ordering exception. +5. One declared late-producer DAG schedules the ACS earnings-universe + materializer, primary PUF/tail pass, all 16 post-clone source operators, + their once-only finalizer, and all 19 bounded transfer groups. Each node + declares every effective input and output. A callback cannot run until each + input is filled on its required scope or has the exact counted + declared-absence receipt which that input contract tolerates. Import + validation rejects unknown producers, ambiguous ownership, uncovered + targets, and cycles, naming a deterministic cycle path. Lexical Kahn waves + make the order independent of registry iteration. The universe node is the + unique first wave, the primary PUF pass is the unique second wave, and later + waves interleave source and transfer work. In particular, PUF batch 5 transfers + `sstb_self_employment_income_before_lsr` before adult care consumes it, PUF + batch 2 transfers `qualified_tuition_expenses` before education consumes it, + pregnancy precedes WIC, and childcare precedes adult care. This order is + derived from producer/input edges, never imposed as a second hand-written + list. Thus the ACS structural-zero rule is also scheduled from declared + inputs rather than hidden inside the primary callback. + + The complete model donor is the ASEC-origin PUF-detail role. Authority is target-specific: every live positive-index clone must already observe a PUF-produced target, every ASEC-origin clone must already observe a source-produced target, and dual-produced targets require the union. A null - on any such producer row is terminal; only the complementary recipient - rows may be filled from QRF predictions. No blanket null-to-zero synthesis - occurs, every producer cell stays byte-identical, and zero residual nulls - are required. -6. The transferred checkpoint records the early gap-fill banks, post-PUF - transfer bank, primary-QRF bank, tail manifest, weights audit, + on any producer row is terminal; only complementary recipient rows may be + filled from QRF predictions. No blanket null-to-zero synthesis occurs, + every producer cell stays byte-identical, and zero residual nulls are + required. +6. The transferred checkpoint records the early gap-fill banks, 19 distinct + late-transfer banks, the primary-QRF bank, the complete 38-node DAG receipt, + tail manifest and its per-status support receipt, weights audit, stack-manifest digest, fraction/seed, clone controls, and the channel-aware - producer-precedence schedule. The same identity regime governs cold and - resumed builds. Checkpoint emission, resume, and final publication each - reject the post-PUF receipt unless it carries the exact canonical stacked - authority; NON-CANONICAL test receipts cannot ship. + producer-precedence schedule. The DAG receipt binds all 71 edges, all input + inventories, six derived waves, exact execution rows, the once-only source + finalizer, and the 19-group/70-target aggregate. Every row hashes the live + content of every declared alternative and output, the callback receipt, and + the preceding row. The top receipt hashes the entry/output frames and chain + terminus. Its digest is anchored in immutable Frame metadata and carried + independently through checkpoints and publication. The same identity + regime governs cold and resumed builds. Checkpoint emission, resume, and + final publication reject a missing, stale, or reissued authority; + NON-CANONICAL test receipts cannot ship. + + Outer materializer v10 also embeds one signed resource-semantics row for + every DAG producer. Static configs are exact; donor tables are bound by the + declared canonical scalar-content codec; primary and transfer banks name + their outer/stage identity derivations; and source receipts name their + callback-receipt digest derivation. The resource keys must exactly equal the + producer's virtual input surface. Registry iteration cannot change the + bytes, and checkpoint discovery positively accepts the current identity but + rejects a stale source asset/config even when every engine and sampling pin + is otherwise identical. 7. Schedule-D preparation, deterministic derivation, seeded inputs, and batched simulation run on the transferred stack. QBI reconciliation uses the same source declaration: it fails on any in-universe self-employment @@ -255,6 +348,10 @@ are allowed only when named by the ACS native-input receipt. emission revalidates the exact structural-rule schema, row arithmetic, per-role proofs, and battery exclusion count from the immutable gate snapshot, so authority metadata cannot be grafted onto invented absence. + Both terminal gates reauthenticate the tail manifest and project its exact + per-status support receipt into gate details. A missing, altered, or rebound + status, observed count, required minimum, attachment decision, or manifest + digest fails closed. At small rungs, comparisons outside the validity domain receipt `insufficient_support`; tolerances do not widen. 9. Only after both gates run does publication write the nullable H5, @@ -263,12 +360,533 @@ are allowed only when named by the ACS native-input receipt. fraction token, seed, code/input/identity pins, phases, gate-receipt pointers, wall time, artifact location, and disposition. +### Late producer/input DAG + +The late stage is a declared producer/input graph, not a fixed source loop +followed by a fixed transfer loop. Its registry contains 38 producers: the ACS +earnings-universe producer, the primary PUF/tail producer, 16 post-clone source +producers, their explicit once-only finalizer, and 19 bounded late-transfer +producers. Import derives +and validates the schedule. Unknown +producers, duplicate ownership, uncovered transfer targets, and cycles fail at +import; a cycle error prints its deterministic cycle path. Readiness is checked +again immediately before each callback. Every required input must be nonnull +on its declared scope, finite when marked numeric, or carry one of that input's +explicitly tolerated counted-absence receipts. A receipt tolerated by one +input does not authorize another input. The readiness fence never converts a +missing value to zero; the separately declared ACS universe producer is the +only named structural-zero materializer. Execution receipts do not trust their +summarized readiness counts: +validation recomputes them from the exact physical-alternative evidence, +requires the exact kind-specific input/output schema, and rejects a completed +producer whose declared output is absent. + +The notation below is executable-contract shorthand. `p`, `tu`, `s`, and `h` +mean person, tax unit, SPM unit, and household. `F(x)` requires numeric finite +values; `+` is an all-of alternative; `|` separates alternatives; and `?R` +means that only the named, counted absence receipt may replace that optional +input. `@weight` is the Frame-resolved entity weight and `@sidecar` or `@bank` +is an authenticated resource receipt, not a physical column. + +The first producer, `acs_pums_earnings_universe`, has this complete ten-row +ACS-scoped inventory: + +```text +F(p.age) +p.person_support_channel +F(p.person_support_clone_index) +p.person_tax_unit_id +p.person_source_id | p.person_id +p.WAGP present +p.SEMP present +F(p.employment_income_before_lsr) ?R +F(p.self_employment_income_before_lsr) ?R +p.@acs_pums_earnings_universe_execution_config +``` + +The two optional mapped numeric rows tolerate only their producer- and +requirement-specific `optional_input:acs_pums_earnings_universe:*` receipts. +The two raw columns must exist, but their legitimate structural nulls remain +part of the source authority. The execution config binds its runtime owner, +the ordered raw-to-mapped column pairs, ACS-only scope, and the complete +universe-rule identity. The producer leaves raw +`WAGP`/`SEMP` untouched, materializes mapped zero only for the declared +under-15 structural universe, and emits both mapped earnings columns plus +`frame.@acs_pums_earnings_universe_application`. Primary PUF consumes all +three outputs directly, making the universe-to-primary edge unavoidable. + +The primary PUF producer has 114 external logical requirements: the following +83-input QRF/tail bundle `Q83`, plus the 31-item validation bundle `V0` below. +`Q83` consists of 17 required core rows, the optional finite +`is_full_time_college_student` tuition fallback, all 56 optional person-output +allocation bases, and all nine optional tax-unit passthroughs. Each optional +row can be absent only under its own counted receipt; a present value must be +finite. `V0` is the common 32-item late-transfer validation bundle `V` with +only the post-PUF clone-attachment manifest removed, because primary PUF +creates that manifest. +The raw ACS `WAGP`/`SEMP` authority, the two ACS-scoped mapped-earnings outputs, +and the application receipt add five direct dependencies, giving the executable +primary contract exactly 119 inputs. + +```text +F(p.age) +filing status = tu.filing_status_input +tax-unit membership = p.person_tax_unit_id + tu.tax_unit_id +F(p.employment_income_before_lsr) +F(p.self_employment_income_before_lsr) +F(p.taxable_interest_income) +dividends = F(p.dividend_income) + | F(p.qualified_dividend_income) + F(p.non_qualified_dividend_income) + | F(tu.dividend_income) +short-term gains = F(p.short_term_capital_gains) + | F(tu.short_term_capital_gains) +long-term gains = F(p.long_term_capital_gains_before_response) + | F(p.long_term_capital_gains) + | F(tu.long_term_capital_gains) +p.person_id +tu.tax_unit_id +p.person_support_channel +p.person_support_clone_index +tu.@weight +F(p.is_full_time_college_student) ?R +tu.@puf_donor_tax_units +tu.@primary_qrf_checkpoint +tu.@primary_puf_execution_config +``` + +The 56 optional finite person allocation bases are, in canonical order: + +```text +employment_income_before_lsr, self_employment_income_before_lsr, +taxable_interest_income, qualified_dividend_income, +non_qualified_dividend_income, tax_exempt_interest_income, +short_term_capital_gains, long_term_capital_gains_before_response, +long_term_capital_gains_on_collectibles, non_sch_d_capital_gains, +taxable_private_pension_income, taxable_ira_distributions, +social_security_retirement, social_security_disability, +social_security_dependents, social_security_survivors, alimony_income, +alimony_expense, salt_refund_income, charitable_cash_donations, +charitable_non_cash_donations, real_estate_taxes, home_mortgage_interest, +investment_interest_expense, investment_income_elected_form_4952, +student_loan_interest, educator_expense, qualified_tuition_expenses, +casualty_loss, unreimbursed_business_employee_expenses, +traditional_ira_contributions_desired, +self_employed_pension_contributions_desired, rental_income, estate_income, +farm_income, farm_operations_income, farm_rent_income, miscellaneous_income, +partnership_income, s_corp_income, +partnership_self_employment_net_earnings, +estate_income_would_be_qualified, +farm_operations_income_would_be_qualified, +farm_rent_income_would_be_qualified, +partnership_s_corp_income_would_be_qualified, +rental_income_would_be_qualified, +self_employment_income_would_be_qualified, +sstb_self_employment_income_would_be_qualified, business_is_sstb, +qualified_bdc_income, qualified_reit_and_ptp_income, +sstb_self_employment_income_before_lsr, +sstb_unadjusted_basis_qualified_property, +sstb_w2_wages_from_qualified_business, +unadjusted_basis_qualified_property, w2_wages_from_qualified_business +``` + +The nine optional finite tax-unit passthroughs are: + +```text +domestic_production_ald, unrecaptured_section_1250_gain, +first_home_mortgage_balance, second_home_mortgage_balance, +first_home_mortgage_interest, second_home_mortgage_interest, +first_home_mortgage_origination_year, +second_home_mortgage_origination_year, health_savings_account_ald +``` + +```text +V0 = support channel + F(clone index) on p, h, tu, s, family, marital_unit + + F(p.person_id) + + F(p.person_household_id) + F(p.person_tax_unit_id) + + F(p.person_spm_unit_id) + F(p.person_family_id) + + F(p.person_marital_unit_id) + + F(h.household_id) + F(tu.tax_unit_id) + F(s.spm_unit_id) + + F(family.family_id) + F(marital_unit.marital_unit_id) + + F(p.person_spine_source_id) + F(p.person_source_id) + + F(h.household_spine_source_id) + F(h.household_source_id) + + F(h.TYPEHUGQ) + h.@weight + + frame.@us_spine_assembly_manifest + + frame.@us_stacked_spine_manifest +``` + +Those are 28 physical provenance/structure columns, one resolved household +weight, and two metadata receipts. Primary PUF declares the same structural +surface, all six resolved-weight resources, 65 PUF/tail columns, and the clone +attachment manifest as outputs, so downstream dependencies are ownership +edges rather than incidental observations. + +The three primary virtual resources are semantic, not row-count assertions. +The donor receipt hashes canonical typed scalar content, ordered columns, and +dtypes. The checkpoint receipt binds the outer routed identity, cache mode, +primary-QRF schema, manifest name, and exact target order; the physical +checkpoint directory basename must independently equal that bound identity. +The execution-config receipt resolves and hashes the actual predictor/output +sequences, all 65 optional allocation/passthrough reads, clone fraction/seed, +QRF seed/estimator count, worker module/interpreter/argv and reviewed fit +environment, strict-recipient and null-preserving doctrines, the once-resolved +aggregate-disaggregation spec and SOI AGI-band bytes/semantics, every tail +selection/topcode/five-times/concentration control, and enabled audit sinks. +The same three receipts form an exact SHA-bound sidecar beside the primary-QRF +manifest; resume refuses a missing or different sidecar, including a +same-row-count donor with changed bytes. This closes stale-bank reuse under a +newly claimed outer route. + +Every one of the 16 source producers consumes the following 16-requirement +wrapper bundle `W`. It is added to the operator-specific kernel inventory in +the table below, even where a kernel requirement names the same physical +column again. The schema-v3 execution config names the operator and binds the +post-clone phase, complete operator registry and contract, declared output +family and formula-owned removals, seed `0`; period `2024`, except housing's +`None`; `force_puf_imputation=True` only for retirement distributions; strict +existing-surface policy; housing QRF controls; and explicit `not_supplied` mode +for the education and weeks-unemployed sidecar arguments. For the 15 +manifest-backed kernels it also hashes the exact packaged `source_stages.json` +bytes, resolved `SourceStageSpec`, and live resolver module/callable, refusing +runtime-helper drift. Thus no callback control or unreachable sidecar +alternative sits outside the registry: + +```text +W = p.@post_clone_source_execution_config + + frame.@us_spine_assembly_manifest + p.PERIDNUM + + F(p.person_support_clone_index) + h.@weight + + F(p.person_id) + F(p.person_household_id) + F(p.person_tax_unit_id) + + F(p.person_spm_unit_id) + F(p.person_family_id) + + F(p.person_marital_unit_id) + + F(h.household_id) + F(tu.tax_unit_id) + F(s.spm_unit_id) + + F(family.family_id) + F(marital_unit.marital_unit_id) +``` + +The common role-aware kernel bundle `C`, used by the source rows marked with +`C`, is: + +```text +F(p.person_id); p.@weight; p.person_support_channel; +F(p.person_support_clone_index) ?R; +F(p.age) | F(p.A_AGE); +F(p.is_male) | F(p.is_female) | F(p.A_SEX); +F(p.has_esi); F(p.person_tax_unit_id); p.tax_unit_role_input; +F(p.employment_income_before_lsr) | F(p.WSAL_VAL); +F(p.self_employment_income_before_lsr) | F(p.SEMP_VAL); +[F(p.social_security_retirement) + F(p.social_security_disability) + + F(p.social_security_survivors) + F(p.social_security_dependents)] + | F(p.SS_VAL); +F(tu.tax_unit_id); tu.filing_status_input | tu.filing_status +``` + +The table gives every kernel input in addition to `W`; `C + ...` expands +exactly to the kernel bundle above. All raw CPS codes and amounts shown in the +table carry `F(...)` finite-numeric semantics unless they are explicitly +domain-checked booleans or strings. An optional receipt is not permission to +excuse a present invalid raw value. + +| Post-clone source producer | Complete effective kernel input set | +|---|---| +| `impute_us_housing_assistance_to_puf_support` | `C + p.person_spm_unit_id + s.spm_unit_id + F(s.receives_housing_assistance) + F(s.takes_up_housing_assistance_if_eligible) + s.spm_unit_support_channel + s.spm_unit_support_clone_index ?R` | +| `with_us_adult_care_inputs` | `F(p.age) + F(p.employment_income_before_lsr) + F(p.self_employment_income_before_lsr) + F(p.sstb_self_employment_income_before_lsr) + F(p.PEDISDRS) + F(p.is_full_time_college_student) + p.tax_unit_role_input + F(p.person_tax_unit_id) + F(p.person_spm_unit_id) + F(p.person_id) + [p.person_support_channel + F(p.person_support_clone_index)] + F(s.spm_unit_pre_subsidy_childcare_expenses) + F(s.spm_unit_id) + F(tu.tax_unit_id) + p.@weight + s.@weight + tu.@weight` | +| `with_us_child_support_inputs` | `C + p.CSP_VAL + p.CHSP_VAL` | +| `with_us_childcare_inputs` | `C + F(p.person_spm_unit_id) + F(p.SPM_CHILDCAREXPNS) + s.spm_unit_id` | +| `with_us_disability_benefits` | `C + p.DIS_VAL1 + p.DIS_SC1 + p.DIS_VAL2 + p.DIS_SC2` | +| `with_us_education_inputs` | `F(p.ED_VAL) + F(p.qualified_tuition_expenses) + p.person_id + p.@weight` | +| `with_us_energy_subsidy_input` | `C + F(p.person_spm_unit_id) + F(p.SPM_ENGVAL) + s.spm_unit_id` | +| `with_us_immigration_inputs` | `p.PRCITSHP + p.PEINUSYR + p.PENATVTY + p.A_AGE + p.A_MARITL + p.A_SPOUSE + p.A_HSCOL + p.WSAL_VAL + p.SEMP_VAL + p.MCARE + p.CAID + p.IHSFLG + p.CHAMPVA + p.MIL + p.PEN_SC1 + p.PEN_SC2 + p.RESNSS1 + p.RESNSS2 + p.SS_YN + p.SSI_YN + p.PEIO1COW + p.A_MJOCC + p.PEAFEVER + p.SPM_CAPHOUSESUB + p.person_id + p.@weight + ([p.source_year + p.source_person_id] | p.person_id)` | +| `with_us_medicare_take_up_input` | `p.MCARE + p.person_id + p.@weight` | +| `with_us_pregnancy_inputs` | `p.A_SEX + p.A_AGE + p.person_id + p.@weight + ([p.source_year + p.source_household_id + p.source_person_id] | p.person_id)` | +| `with_us_prior_year_income_inputs` | `C + F(p.source_year) + F(p.PERIDNUM) + F(p.WSAL_VAL) + F(p.SEMP_VAL) + F(p.I_ERNVAL) + F(p.I_SEVAL) + F(p.employment_income_last_year) + F(p.self_employment_income_last_year)` | +| `with_us_retirement_contribution_inputs` | `C + p.RETCB_VAL + p.WSAL_VAL + p.SEMP_VAL` | +| `with_us_retirement_distribution_inputs` | `C + p.DST_SC1 + p.DST_VAL1 + p.DST_SC2 + p.DST_VAL2 + p.DST_SC1_YNG + p.DST_VAL1_YNG + p.DST_SC2_YNG + p.DST_VAL2_YNG + p.taxable_ira_distributions` | +| `with_us_weeks_unemployed` | `F(p.source_year) + F(p.PERIDNUM) + F(p.LKWEEKS) + (F(p.age) | F(p.A_AGE)) + (F(p.is_male) | F(p.is_female) | F(p.A_SEX)) + (F(p.tax_unit_is_joint) | [p.person_tax_unit_id + tu.tax_unit_id + tu.filing_status_input] | [p.person_tax_unit_id + tu.tax_unit_id + tu.filing_status]) + (p.tax_unit_role_input | [F(p.is_tax_unit_head) + F(p.is_tax_unit_spouse) + F(p.is_tax_unit_dependent)]) + (F(p.unemployment_compensation) | F(p.UC_VAL)) ?R + p.person_support_channel + p.@weight` | +| `with_us_wic_claim_input` | `F(p.age) + F(p.is_female) + F(p.is_pregnant) + F(p.own_children_in_household) + F(p.person_family_id) + p.@weight + ([p.source_year + p.source_household_id + p.source_person_id] | p.person_support_source_id | p.person_id)` | +| `with_us_workers_compensation` | `C + p.WC_VAL` | + +The table plus `W` is the complete external logical inventory. The executable +contract additionally carries direct evidence from every declared producer of +an inventory column, plus the cross-source dependencies named in the edge +table. These counts make that expansion auditable: + +| Source producer | External inventory rows | Executable contract inputs | +|---|---:|---:| +| `impute_us_housing_assistance_to_puf_support` | 36 | 59 | +| `with_us_adult_care_inputs` | 33 | 54 | +| `with_us_child_support_inputs` | 32 | 53 | +| `with_us_childcare_inputs` | 33 | 54 | +| `with_us_disability_benefits` | 34 | 55 | +| `with_us_education_inputs` | 20 | 35 | +| `with_us_energy_subsidy_input` | 33 | 54 | +| `with_us_immigration_inputs` | 43 | 57 | +| `with_us_medicare_take_up_input` | 19 | 33 | +| `with_us_pregnancy_inputs` | 21 | 35 | +| `with_us_prior_year_income_inputs` | 38 | 59 | +| `with_us_retirement_contribution_inputs` | 33 | 54 | +| `with_us_retirement_distribution_inputs` | 39 | 61 | +| `with_us_weeks_unemployed` | 26 | 41 | +| `with_us_wic_claim_input` | 23 | 38 | +| `with_us_workers_compensation` | 31 | 52 | + +The 17th source-side node is the explicit `source_finalizer`. Its complete +17-input set is the 16 virtual resources +`p.@source_receipt:`, one for every table row above, plus +`p.@source_finalizer_execution_config`. Each source resource hashes the exact +corresponding callback receipt. The schema-v2 finalizer config binds the phase, +source registry, formula-owned exclusions, complete deferred-input declarations, +and deferred status. Only after all 17 exist may the finalizer materialize the +three deliberately deferred SCF columns +`bank_account_assets`, `bond_assets`, and `stock_assets` with their declared +absence receipts. This makes finalization a DAG node rather than a hidden +mutation after the schedule. + +Every transfer consumes the 46-row external logical inventory `V + T(E)` plus +direct producer evidence for every primary/source-owned physical input and +target in the next table. The adult-care transfer alone adds the required +47th logical row `p.tax_unit_role_input`, consumed by its deterministic +post-fit reconciliation. `V` is the exact common validation surface: 28 +physical columns, the resolved household weight, and three immutable metadata +receipts. + +```text +V = support channel + F(clone index) on p, h, tu, s, family, marital_unit + + F(p.person_id) + + F(p.person_household_id) + F(p.person_tax_unit_id) + + F(p.person_spm_unit_id) + F(p.person_family_id) + + F(p.person_marital_unit_id) + + F(h.household_id) + F(tu.tax_unit_id) + F(s.spm_unit_id) + + F(family.family_id) + F(marital_unit.marital_unit_id) + + F(p.person_spine_source_id) + F(p.person_source_id) + + F(h.household_spine_source_id) + F(h.household_source_id) + + F(h.TYPEHUGQ) + h.@weight + + frame.@us_spine_assembly_manifest + + frame.@us_stacked_spine_manifest + + frame.@us_puf_clone_attachment_manifest +``` + +For a transfer whose target entity is `E`, the complete 14-requirement model +and weight bundle `T(E)` is: + +```text +T(E) = F(p.age) + F(p.is_female) + p.@weight + E.@weight + + E.@late_transfer_model_config + E.@late_transfer_target_bank + + [F(p.state_fips) + | (F(p.person_household_id) + F(h.household_id) + F(h.state_fips))] + + F(p.employment_income_before_lsr) ?R + + F(p.self_employment_income_before_lsr) ?R + + [(F(p.social_security_retirement) + + F(p.social_security_disability) + + F(p.social_security_dependents) + + F(p.social_security_survivors)) + | F(p.acs_social_security_income)] ?R + + [(F(p.taxable_private_pension_income) + + F(p.tax_exempt_private_pension_income) + + F(p.taxable_ira_distributions)) + | F(p.acs_retirement_income)] ?R + + [(F(p.taxable_interest_income) + F(p.tax_exempt_interest_income) + + F(p.qualified_dividend_income) + + F(p.non_qualified_dividend_income) + F(p.rental_income) + + F(p.estate_income)) + | F(p.acs_interest_dividend_rental_income)] ?R + + (F(p.is_household_head) | F(p.RELSHIPP) | F(p.A_EXPRRP) + | F(p.A_LINENO)) ?R + + (p.tenure_type | s.spm_unit_tenure_type | F(h.TEN) + | F(h.H_TENURE)) ?R +``` + +Every one of the 19 transfer nodes also requires direct primary-PUF evidence +for every primary-owned physical alternative in `V + T(E)`, including +`p.tax_exempt_interest_income` and `p.estate_income`, plus direct producer +evidence for every target listed below. A target shown in both producer columns +requires both scopes; that is the two-target PUF/source overlap. Thus the table +is the complete per-node producer delta over `V + T(E)`, as well as the exact +70-target partition. Transfer rows abbreviate the registry's leading +`transfer:`; source names in these tables abbreviate the leading `source:`. + +For every transfer, schema-v3 `@late_transfer_model_config` binds that node's +name, entity, family, ordered targets, seed, estimator count, canonical maximum +targets per fit, required/optional predictor order, combined-feature mappings, +target codecs, housing-head and tenure precedence/codes, immigration codec, +and deterministic post-fit structure. Bounded DAG groups explicitly disable +the opportunistic Schedule-D derivation; the later whole-pool tax-unit derive +operator is its sole owner. Adult care keeps its declared reconciliation and +therefore gates on `tax_unit_role_input`. `@late_transfer_target_bank` binds +either the durable bank's +validated identity SHA-256 or the explicit `ephemeral_no_checkpoint` mode; a +non-null bank without an identity is rejected before dispatch. Each virtual +receipt has an exact kind-specific inner schema and its own SHA-256, and its +execution-row input evidence must hash the identical receipt. The source +finalizer applies the same rule to each of its sixteen source-receipt inputs. + +| Transfer producer | Targets | PUF target inputs | Source target inputs | +|---|---|---|---| +| `person/adult_care` | `is_incapable_of_self_care`, `pre_subsidy_care_expenses` | — | both from `with_us_adult_care_inputs` | +| `person/model_required_boolean` | `is_pregnant` | — | from `with_us_pregnancy_inputs` | +| `person/puf_tax_itemization__batch_1` | `tax_exempt_interest_income`, `long_term_capital_gains_on_collectibles`, `non_sch_d_capital_gains`, `alimony_expense`, `salt_refund_income`, `charitable_cash_donations`, `charitable_non_cash_donations`, `home_mortgage_interest` | all targets | — | +| `person/puf_tax_itemization__batch_2` | `investment_interest_expense`, `investment_income_elected_form_4952`, `student_loan_interest`, `educator_expense`, `qualified_tuition_expenses`, `casualty_loss`, `unreimbursed_business_employee_expenses`, `traditional_ira_contributions_desired` | all targets | `traditional_ira_contributions_desired` from `with_us_retirement_contribution_inputs` | +| `person/puf_tax_itemization__batch_3` | `self_employed_pension_contributions_desired`, `estate_income`, `farm_income`, `farm_rent_income`, `partnership_income`, `partnership_self_employment_net_earnings`, `estate_income_would_be_qualified`, `farm_operations_income_would_be_qualified` | all targets | `self_employed_pension_contributions_desired` from `with_us_retirement_contribution_inputs` | +| `person/puf_tax_itemization__batch_4` | `farm_rent_income_would_be_qualified`, `partnership_s_corp_income_would_be_qualified`, `rental_income_would_be_qualified`, `self_employment_income_would_be_qualified`, `sstb_self_employment_income_would_be_qualified`, `business_is_sstb`, `qualified_bdc_income`, `qualified_reit_and_ptp_income` | all targets | — | +| `person/puf_tax_itemization__batch_5` | `sstb_self_employment_income_before_lsr`, `sstb_unadjusted_basis_qualified_property`, `sstb_w2_wages_from_qualified_business`, `unadjusted_basis_qualified_property`, `w2_wages_from_qualified_business` | all targets | — | +| `person/source_operator_child_support` | `child_support_expense`, `child_support_received` | — | both from `with_us_child_support_inputs` | +| `person/source_operator_disability_benefits` | `disability_benefits` | — | from `with_us_disability_benefits` | +| `person/source_operator_education_inputs` | `attends_eligible_educational_institution_for_american_opportunity_credit`, `educational_assistance`, `has_american_opportunity_credit_1098_t_or_exception`, `has_american_opportunity_credit_institution_ein`, `is_enrolled_at_least_half_time_for_american_opportunity_credit`, `is_pursuing_credential_for_american_opportunity_credit` | — | all from `with_us_education_inputs` | +| `person/source_operator_immigration` | `ssn_card_type`, `immigration_status_str` | — | both from `with_us_immigration_inputs` | +| `person/source_operator_medicare_take_up` | `takes_up_medicare_if_eligible` | — | from `with_us_medicare_take_up_input` | +| `person/source_operator_retirement_contributions` | `roth_401k_contributions_desired`, `roth_ira_contributions_desired`, `traditional_401k_contributions_desired` | — | all from `with_us_retirement_contribution_inputs` | +| `person/source_operator_retirement_distributions` | `keogh_distributions`, `tax_exempt_ira_distributions`, `taxable_401k_distributions`, `taxable_403b_distributions`, `taxable_sep_distributions` | — | all from `with_us_retirement_distribution_inputs` | +| `person/source_operator_weeks_unemployed` | `weeks_unemployed` | — | from `with_us_weeks_unemployed` | +| `person/source_operator_wic_claim` | `would_claim_wic` | — | from `with_us_wic_claim_input` | +| `person/source_operator_workers_compensation` | `workers_compensation` | — | from `with_us_workers_compensation` | +| `tax_unit/puf_tax_itemization` | `domestic_production_ald`, `unrecaptured_section_1250_gain`, `first_home_mortgage_balance`, `first_home_mortgage_interest`, `first_home_mortgage_origination_year`, `health_savings_account_ald` | all targets | — | +| `spm_unit/source_operator_energy_subsidy` | `spm_unit_energy_subsidy` | — | from `with_us_energy_subsidy_input` | + +The resulting executable transfer contracts contain 92–100 inputs; 46 is the +common logical inventory, not the complete contract count: + +| Transfer producer | Executable contract inputs | +|---|---:| +| `person/adult_care` | 94 | +| `person/model_required_boolean` | 92 | +| `person/puf_tax_itemization__batch_1` | 98 | +| `person/puf_tax_itemization__batch_2` | 100 | +| `person/puf_tax_itemization__batch_3` | 99 | +| `person/puf_tax_itemization__batch_4` | 99 | +| `person/puf_tax_itemization__batch_5` | 96 | +| `person/source_operator_child_support` | 93 | +| `person/source_operator_disability_benefits` | 92 | +| `person/source_operator_education_inputs` | 97 | +| `person/source_operator_immigration` | 93 | +| `person/source_operator_medicare_take_up` | 92 | +| `person/source_operator_retirement_contributions` | 94 | +| `person/source_operator_retirement_distributions` | 96 | +| `person/source_operator_weeks_unemployed` | 92 | +| `person/source_operator_wic_claim` | 92 | +| `person/source_operator_workers_compensation` | 92 | +| `spm_unit/source_operator_energy_subsidy` | 93 | +| `tax_unit/puf_tax_itemization` | 98 | + +#### Complete dependency edges + +The following grouped tables enumerate all 71 unique producer-to-consumer edges. +Multiple values on one row are the input reasons carried by that edge. Bare +source names carry the registry prefix `source:` and transfer paths carry +`transfer:`. + +The first edge is `acs_pums_earnings_universe -> primary_puf_qrf`, carried by +the two ACS-scoped mapped earnings columns and the whole-frame universe +application receipt. + +Every primary-PUF-to-source edge carries the same 14-item ownership base `S14`: + +```text +S14 = p.person_support_clone_index + p.person_id + + p.person_household_id + p.person_tax_unit_id + p.person_spm_unit_id + + p.person_family_id + p.person_marital_unit_id + + h.household_id + tu.tax_unit_id + s.spm_unit_id + + family.family_id + marital_unit.marital_unit_id + + p.@weight + h.@weight +``` + +The following table gives the complete additions to `S14` on each of the 16 +primary-PUF-to-source edges: + +| Consumer source | Additional inputs supplied by primary PUF | +|---|---| +| `impute_us_housing_assistance_to_puf_support` | person support channel; employment; self-employment; four Social Security components; SPM-unit support channel and clone index | +| `with_us_adult_care_inputs` | person support channel; employment; self-employment; SPM-unit and tax-unit weights | +| `with_us_child_support_inputs` | person support channel; employment; self-employment; four Social Security components | +| `with_us_childcare_inputs` | person support channel; employment; self-employment; four Social Security components | +| `with_us_disability_benefits` | person support channel; employment; self-employment; four Social Security components | +| `with_us_education_inputs` | — | +| `with_us_energy_subsidy_input` | person support channel; employment; self-employment; four Social Security components | +| `with_us_immigration_inputs` | — | +| `with_us_medicare_take_up_input` | — | +| `with_us_pregnancy_inputs` | — | +| `with_us_prior_year_income_inputs` | person support channel; employment; self-employment; four Social Security components | +| `with_us_retirement_contribution_inputs` | person support channel; employment; self-employment; four Social Security components | +| `with_us_retirement_distribution_inputs` | person support channel; employment; self-employment; four Social Security components; `taxable_ira_distributions` | +| `with_us_weeks_unemployed` | person support channel | +| `with_us_wic_claim_input` | — | +| `with_us_workers_compensation` | person support channel; employment; self-employment; four Social Security components | + +There are also 19 primary-PUF-to-transfer edges: one to every row of the +transfer table above. `B(E)` is the exact set of primary-owned physical +alternatives in the published `V + T(E)` inventory plus the shared PUF-clone +predictors `tax_exempt_interest_income` and `estate_income`. `B(person)` has 45 +inputs; `B(tax_unit)` and `B(spm_unit)` each have 46 because their target +weight is distinct from the person weight. Every primary-to-transfer edge +carries `B(E)` plus each PUF-owned target in that row which is not already in +`B(E)`. The transfer target table and its contract counts therefore enumerate +the complete input reasons for all 19 edges. + +The remaining 19 edges are: + +| Producer | Consumer | Input reason | +|---|---|---| +| `with_us_adult_care_inputs` | `transfer:person/adult_care` | `is_incapable_of_self_care`, `pre_subsidy_care_expenses` | +| `with_us_child_support_inputs` | `transfer:person/source_operator_child_support` | `child_support_expense`, `child_support_received` | +| `with_us_childcare_inputs` | `with_us_adult_care_inputs` | `spm_unit_pre_subsidy_childcare_expenses` | +| `with_us_disability_benefits` | `transfer:person/source_operator_disability_benefits` | `disability_benefits` | +| `with_us_education_inputs` | `transfer:person/source_operator_education_inputs` | six education outputs listed above | +| `with_us_energy_subsidy_input` | `transfer:spm_unit/source_operator_energy_subsidy` | `spm_unit_energy_subsidy` | +| `with_us_immigration_inputs` | `transfer:person/source_operator_immigration` | `ssn_card_type`, `immigration_status_str` | +| `with_us_medicare_take_up_input` | `transfer:person/source_operator_medicare_take_up` | `takes_up_medicare_if_eligible` | +| `with_us_pregnancy_inputs` | `with_us_wic_claim_input` | `is_pregnant` | +| `with_us_pregnancy_inputs` | `transfer:person/model_required_boolean` | `is_pregnant` | +| `with_us_retirement_contribution_inputs` | `transfer:person/puf_tax_itemization__batch_2` | `traditional_ira_contributions_desired` source scope | +| `with_us_retirement_contribution_inputs` | `transfer:person/puf_tax_itemization__batch_3` | `self_employed_pension_contributions_desired` source scope | +| `with_us_retirement_contribution_inputs` | `transfer:person/source_operator_retirement_contributions` | three contribution outputs listed above | +| `with_us_retirement_distribution_inputs` | `transfer:person/source_operator_retirement_distributions` | five distribution outputs listed above | +| `with_us_weeks_unemployed` | `transfer:person/source_operator_weeks_unemployed` | `weeks_unemployed` | +| `with_us_wic_claim_input` | `transfer:person/source_operator_wic_claim` | `would_claim_wic` | +| `with_us_workers_compensation` | `transfer:person/source_operator_workers_compensation` | `workers_compensation` | +| `transfer:person/puf_tax_itemization__batch_2` | `with_us_education_inputs` | `qualified_tuition_expenses` | +| `transfer:person/puf_tax_itemization__batch_5` | `with_us_adult_care_inputs` | `sstb_self_employment_income_before_lsr` | + +Finally, there are 16 source-to-finalizer edges: each of the 16 source +producers in the source-input table has one edge to `source_finalizer`, carried +by its exact `p.@source_receipt:` resource. Thus the exhaustive count +is 1 universe-to-primary + 16 primary-to-source + 19 primary-to-transfer + 19 +cross/source-to-transfer + 16 source-to-finalizer = 71. + +The lexically canonical waves have sizes `(1, 1, 17, 14, 3, 2)`: + +1. `acs_pums_earnings_universe`. +2. `primary_puf_qrf`. +3. Housing assistance; child support; childcare; disability; energy; + immigration; Medicare; pregnancy; prior-year income; retirement + contributions; retirement distributions; weeks unemployed; workers' + compensation; person PUF batches 1, 4, and 5; tax-unit PUF transfer. +4. Adult care; WIC; pregnancy transfer; person PUF batches 2 and 3; child + support, disability, immigration, Medicare, retirement-contribution, + retirement-distribution, weeks-unemployed, workers'-compensation, and + SPM-energy transfers. +5. Education; adult-care transfer; WIC transfer. +6. `source_finalizer` and education transfer. + +Registry schema version 13 and execution-receipt schema version 3 bind the +canonical input declarations, outputs, edges, waves, exact kind-specific +virtual-resource bindings, content-hashed execution-row schema, and immutable +transition authority. The schedule SHA-256 is +`dbae9f945966a58592915780be78137e011d060271af6c933870a55db297baab`; +the full payload SHA-256 is +`95ee19cd1b4d1cf321a32910c234ebc460aa47f9cc30e03fa8560ea6ae5e2eb8`. +Reversing registry iteration produces those same bytes. + +The virtual-resource payload ledger is independently versioned: ACS-universe +config v2, primary execution config v3, source execution config v3, source +finalizer config v2, and transfer model config v3. Donor content, +primary-checkpoint routing, source callback receipts, and transfer-bank routing +remain v1. The outer all-producer resource-semantics receipt is v1 and binds +both these static schemas and every dynamic derivation mode. + ### Downstream hard-completeness audit This table makes the stacked 1% supplier and starvation behavior explicit at every remaining boundary. An early `unmodeled_rows` receipt is merely an -accounting result; `insufficient_support` is a later battery status reached -only after a comparison surface is complete and valid. +accounting result. The tail stage may issue its declared per-status +`insufficient_support` receipt before terminal evaluation; the by-origin +battery uses the same status name only after a comparison surface is complete +and valid. Neither receipt authorizes an upstream null. | Boundary | Hard requirement | Stacked 1% supplier | Can an upstream insufficient-support/unmodeled state starve it? | |---|---|---|---| @@ -277,13 +895,13 @@ only after a comparison surface is complete and valid. | PUF raw predictor sources | Every filing-status, count, and income component is observed in its declared source universe. Raw WAGP/SEMP authority is present and agrees with mapped leaves; a cross-grain source collision is rejected. A null on any eligible member fails before coercion. | Structure supplies status/count; ACS-native or ASEC-carried earnings supply earnings; early transfer supplies interest, dividends, and gains. | No. ACS under-15 WAGP/SEMP blanks are an exact source-universe state, not transfer starvation; all other source nulls fail. | | PUF tax-unit features | Every clone-1 recipient has a finite feature vector. Post-aggregation NaN, `+inf`, and `-inf` are counted by named predictor and rejected before fitting; none is coerced or snapped to zero. | Universe-aware person sums plus tax-unit structural inputs. | No. Eligible member values must be complete; the only special case is an all-child unit whose numeric-zero predictor is explicitly owned and counted by the named universe-zero rule. | | Primary QRF banks and chain | Donor/recipient banks are immutable; target order and RNG prefix are contiguous; all targets complete; live recipient identity, source-universe receipt, and feature digest match before finalization. | The processed full PUF donor and strict recipient checkpoint initialized above. | No. Mutation or missing receipt invalidates the bank; it cannot resume under legacy semantics. | -| Outer pool checkpoint identity and resume | Schema/materializer/authority v6 plus the ACS-universe and QBI-mutation contract identities must match exactly before any cached stage is discovered. | Fresh input pins, live stack receipt, scale controls, code identity, and both semantic contract identities. | No. Every v1--v5 root, target, materializer, or authority payload is stale; a self-consistent old receipt cannot reopen a checkpoint. | -| Clone-2 capital-gains tail | Candidate recipients have the required filing-status/AGI support, positive donor mass, unique household lineage, and sufficient weight capacity; every selected donor is assigned once. | Completed clone-1 QRF output and full PUF tail donors. | No early residual is accepted. Universe-aware PUF recipients remain eligible, including explicitly receipted empty-universe tax units. | -| Post-clone source completion | Each source operator preserves structure and emits its declared ASEC-evidenced outputs; unavailable peer cells remain null only until late transfer. | ASEC evidence rows plus completed PUF clone outputs. | Temporarily: peer nulls are intentional here, but the next zero-residual transfer must consume them. | -| Post-PUF transfer | Every declared PUF-clone or ASEC source-producer cell is nonnull; all complementary recipients are filled; the allowed count for both unmodeled and residual rows is zero. | Forty-three PUF and 30 source targets, with three overlaps, supply the 70-target late surface. | No. A missing producer or recipient value is terminal at this boundary. | +| Outer pool checkpoint identity and resume | Primary-QRF schema v6, tail-manifest schema v2, late-registry schema v14/receipt schema v3, outer stacked materializer v10/authority v9, stacked pool-stage materializer v5, pool manifest schema v7, and the ACS-universe, QBI-mutation, tail-support, late-DAG, and signed virtual-resource-semantics identities must match exactly before any cached stage is discovered. The retiring legacy envelope remains manifest schema v4/materializer v3. | Fresh input pins, live stack receipt, scale controls, code identity, and all semantic contract identities. | No. An older stacked materializer or authority payload is stale; a self-consistent old receipt cannot reopen a checkpoint. Primary-QRF v6 remains current. | +| Clone-2 capital-gains tail | Each filing status requires as many eligible recipient households as selected q99.5 donors. Eligibility requires unique single-tax-unit PUF-detail lineage and half-weight capacity for the global maximum assigned donor weight. An adequate status assigns every selected donor once; a thin status skips as a whole with a named, counted `insufficient_support` receipt. | Completed clone-1 QRF output and full PUF tail donors. At 1%, `SINGLE` and `HEAD_OF_HOUSEHOLD` attach, `JOINT` and `SEPARATE` skip, and zero-requirement `SURVIVING_SPOUSE` is `not_applicable`. | No widening or partial attachment is permitted. All 22 AGI bands provide nearest-first fallback only inside a status. Universe-aware PUF recipients remain eligible, including explicitly receipted empty-universe tax units. | +| Late producer DAG | Before any callback, all declared inputs are filled on their required scopes or carry an input-specific counted absence receipt; numeric inputs are finite. The exact derived order, readiness rows, once-only source finalizer, and bounded transfer receipts must validate. | ACS earnings-universe materialization, primary PUF/tail, 16 source producers, and 19 bounded transfer groups execute in six derived waves. | No. The refusing producer names the unfilled input and its declared producing stage. A cycle fails at import with its path. | +| Late transfer completion | Every declared PUF-clone or ASEC source-producer cell is nonnull; all complementary recipients are filled; the allowed count for both unmodeled and residual rows is zero. | Forty-three PUF and 29 source targets, with two overlaps, supply the 70-target late surface. | No. A missing producer or recipient value is terminal at this boundary. | | Fit-weight audit | Every primary and post-PUF QRF fit receipts its resolved entity weight kind, and the collected fit records pass the weights audit before a transferred checkpoint can exist. | Calibrated household weights mapped by the frame to each modeled entity. | No. A missing, inconsistent, or manually substituted weight declaration fails before checkpoint emission. | -| Tail preservation | Tail manifest, descendants, IDs, weights, provenance, joint vector, and non-tail QRF cells remain exact after completion, transfer, derive, seed, and simulation. | The tail manifest bound during the PUF pass. | Completeness receipts cannot authorize a mutation; any byte or identity change fails. | -| Schedule-D derive | Both transferred parent columns are finite for every person and align to every tax unit. | Completed post-PUF transfer plus tail replacements. | No. A residual would fail late transfer first and derive again by name. | +| Tail preservation | Tail manifest, support decisions, attached descendants, IDs, weights, provenance, joint vector, and non-tail QRF cells remain exact after completion, transfer, derive, seed, and simulation. | The schema-v2 tail manifest and support receipt bound during the PUF pass and projected into both terminal gates. | A support receipt cannot authorize mutation. Any byte or identity change in an attached status, any descendant for a skipped status, or any receipt change fails. | +| Schedule-D derive | Both transferred parent columns are finite for every person and align to every tax unit. Bounded late-transfer groups do not write this leaf; the whole-pool tax-unit derive is its sole canonical owner. | Completed late transfer plus tail replacements. | No. A residual would fail late transfer first and derive again by name. | | QBI derive | All QBI detail outputs are finite; self-employment is finite wherever its source applies; every independent archived QBI identity holds. The declared surface includes the base self-employment rewrite and binds pre/post digests. Its exact receipt is recomputed and authenticated at every persisted and publication boundary. | PUF/source detail plus ACS/ASEC native self-employment. Raw under-15 ACS `SEMP` remains structurally blank; mapped `self_employment_income_before_lsr` is a named, receipted universe zero. | No silent starvation. Every mapped ACS under-15 base value is held at its receipted universe zero across clone roles; all derived QBI cells remain in scope, and an in-universe null, forged receipt, or non-kernel output fails. | | Take-up seed | Every administratively seeded variable completes; transfer-owned take-up cannot use a default; only explicitly non-transfer-owned inputs may use receipted engine defaults. | Seed kernels, the complete transfer surface, and declared defaults. | Transfer-owned residuals fail. A declared default is a separate modeled state, not an insufficient-support receipt. | | SSI simulation projection | Every nullable engine input has a declared default on the disposable projection; the engine returns exactly one SSI value per person. | The persistent derived/seeded pool plus separately receipted ephemeral defaults. | A projection default can enable simulation but cannot cure the persistent pool; terminal evaluation returns to the original inputs plus SSI. | @@ -293,15 +911,19 @@ only after a comparison surface is complete and valid. | Manifest construction and canonical publication closure | Legacy and stacked builders reauthenticate QBI live output, canonical stacked authority, terminal-gate receipts, H5/diagnostics run IDs, and artifact digests before readiness can be asserted. | The validated persistent pool, immutable stage receipts, terminal gate snapshot, and atomically staged publication files. | No. Construction rejects forged or wrong-route receipts; publication begins with a non-ready tombstone, and only one fully authenticated run can replace it with a ready manifest. | The audit leaves no generic “receipted but null” path into a hard consumer. -Structural absence is target- and universe-exact; sample-size support affects -only whether an otherwise complete terminal comparison is testable. +Structural absence is target- and universe-exact. Sample-size support affects +only whether a complete filing-status tail can attach and whether an otherwise +complete terminal comparison is testable. ### Retiring `--legacy-two-spine` sequence The explicit compatibility flag preserves the previous assemble-first pool path, including its publication bytes. It remains reproducible for lineage comparison but is not the production default. That path runs this fixed -sequence: +sequence under its preserved manifest schema 4/checkpoint materializer 3 +identity. The loader uses the complete schema/envelope surface and rejects a +stacked artifact whose pipeline markers were stripped, so schema lowering +cannot bypass late-DAG validation: 1. `assemble_spines({"asec": ..., "acs": ...})` creates the first shared population state and binds the immutable assembly receipt. @@ -436,9 +1058,11 @@ source ingestion and faithful schema harmonization -> uniformly sample both survey arms and assemble one stack -> prepare native predictors -> banked cross-origin gap-fill - -> one PUF QRF pass plus clone-2 capital-gains tail - -> source completion - -> banked post-PUF transfer of newly materialized targets + -> derived 38-node late producer DAG: + ACS earnings-universe materialization + -> PUF QRF plus clone-2 capital-gains tail + -> interleaved source completion and 19 bounded transfer groups + -> exact source finalization and transfer aggregation -> derive -> seed take-up and other stochastic inputs -> simulate diff --git a/packages/microcosm-build/src/microcosm/build/frame_checkpoint.py b/packages/microcosm-build/src/microcosm/build/frame_checkpoint.py index 5334aadf..ac26729c 100644 --- a/packages/microcosm-build/src/microcosm/build/frame_checkpoint.py +++ b/packages/microcosm-build/src/microcosm/build/frame_checkpoint.py @@ -31,6 +31,7 @@ MassChangeRecord, WeightKind, Weights, + nullable_boolean_values_and_mask, ) __all__ = [ @@ -40,7 +41,11 @@ "write_frame_checkpoint", ] -FRAME_CHECKPOINT_SCHEMA_VERSION = 2 +FRAME_CHECKPOINT_SCHEMA_VERSION = 3 +_LEGACY_FRAME_CHECKPOINT_SCHEMA_VERSION = 2 +_SUPPORTED_FRAME_CHECKPOINT_SCHEMA_VERSIONS = frozenset( + {_LEGACY_FRAME_CHECKPOINT_SCHEMA_VERSION, FRAME_CHECKPOINT_SCHEMA_VERSION} +) _ARTIFACT_KIND = "populace_frame_checkpoint" _ROOT = "_populace_frame_checkpoint" @@ -50,6 +55,7 @@ _ENCODING_DATETIME = "datetime64" _ENCODING_TIMEDELTA = "timedelta64" _ENCODING_OBJECT = "object_scalars_v1" +_ENCODING_NULLABLE_BOOLEAN = "nullable_boolean_v1" _TAG_NONE = 0 _TAG_PD_NA = 1 @@ -339,10 +345,26 @@ def _checkpoint_metadata( } ) + strata_spec = _series_spec(frame.strata, label="strata") + uses_nullable_boolean = any( + column.get("encoding") == _ENCODING_NULLABLE_BOOLEAN + for table in table_specs + for column in table["columns"] + ) or any( + table["index"].get("encoding") == _ENCODING_NULLABLE_BOOLEAN + for table in table_specs + ) + uses_nullable_boolean = uses_nullable_boolean or ( + strata_spec.get("encoding") == _ENCODING_NULLABLE_BOOLEAN + ) schema = frame.schema return { "artifact_kind": _ARTIFACT_KIND, - "schema_version": FRAME_CHECKPOINT_SCHEMA_VERSION, + "schema_version": ( + FRAME_CHECKPOINT_SCHEMA_VERSION + if uses_nullable_boolean + else _LEGACY_FRAME_CHECKPOINT_SCHEMA_VERSION + ), "schema": { "person_entity": schema.person_entity, "group_entities": list(schema.group_entities), @@ -368,7 +390,7 @@ def _checkpoint_metadata( if "household" in frame.weighted_entities else None ), - "strata": _series_spec(frame.strata, label="strata"), + "strata": strata_spec, "mass_log": [_mass_change_payload(record) for record in frame.mass_log], "external_metadata": external_metadata, } @@ -380,7 +402,7 @@ def _frame_tables(frame: Frame) -> tuple[tuple[str, pd.DataFrame], ...]: return (*entities, *links) -def _series_spec(series: pd.Series, *, label: str) -> dict[str, str]: +def _series_spec(series: pd.Series, *, label: str) -> dict[str, object]: dtype = series.dtype if isinstance(dtype, pd.CategoricalDtype): raise TypeError( @@ -392,6 +414,12 @@ def _series_spec(series: pd.Series, *, label: str) -> dict[str, str]: f"Frame checkpoint does not support timezone-aware dtype for {label!r}; " "convert it to timezone-naive datetime64 first." ) + if isinstance(dtype, pd.BooleanDtype): + return { + "dtype": "boolean", + "encoding": _ENCODING_NULLABLE_BOOLEAN, + "has_null_mask": bool(series.isna().any()), + } if isinstance(dtype, pd.api.extensions.ExtensionDtype) and not isinstance( dtype, pd.StringDtype ): @@ -523,6 +551,16 @@ def _write_series(group: Any, series: pd.Series, spec: Mapping[str, object]) -> _write_numpy_dataset(group, "offsets", offsets) _write_bytes_dataset(group, "payload", payload) return + if encoding == _ENCODING_NULLABLE_BOOLEAN: + values, null_mask = nullable_boolean_values_and_mask(series) + _write_numpy_dataset(group, "values", values) + if spec.get("has_null_mask") is True: + _write_numpy_dataset( + group, + "null_mask", + null_mask.astype(np.uint8, copy=False), + ) + return raise RuntimeError(f"Unknown checkpoint series encoding {encoding!r}.") @@ -552,6 +590,57 @@ def _read_series( payload = _read_bytes_dataset(group, "payload", path) values = _decode_object_values(offsets, payload, path=path, label=label) series = pd.Series(values, dtype=restore_dtype, copy=False) + elif encoding == _ENCODING_NULLABLE_BOOLEAN: + if dtype != "boolean": + raise ValueError( + f"Frame checkpoint {path} {label!r} nullable boolean encoding " + f"has declared dtype {dtype!r}, not 'boolean'." + ) + has_null_mask = spec.get("has_null_mask") + if type(has_null_mask) is not bool: + raise ValueError( + f"Frame checkpoint {path} {label!r} nullable boolean " + "has_null_mask must be a boolean." + ) + values = _read_numpy_dataset(group, "values", path) + if values.ndim != 1 or values.dtype != np.dtype(np.bool_): + raise ValueError( + f"Frame checkpoint {path} {label!r} nullable boolean values " + "must be a one-dimensional bool array." + ) + if has_null_mask: + if "null_mask" not in group: + raise ValueError( + f"Frame checkpoint {path} {label!r} is missing its null mask." + ) + null_mask = _read_numpy_dataset(group, "null_mask", path) + if ( + null_mask.ndim != 1 + or null_mask.dtype != np.dtype(np.uint8) + or len(null_mask) != len(values) + or ((null_mask != 0) & (null_mask != 1)).any() + or not null_mask.any() + ): + raise ValueError( + f"Frame checkpoint {path} {label!r} null mask must be a " + "one-dimensional uint8 0/1 array aligned to its values." + ) + mask = null_mask.astype(np.bool_, copy=False) + if values[mask].any(): + raise ValueError( + f"Frame checkpoint {path} {label!r} null mask covers " + "noncanonical true storage bits." + ) + else: + if "null_mask" in group: + raise ValueError( + f"Frame checkpoint {path} {label!r} has an unexpected null mask." + ) + mask = np.zeros(len(values), dtype=np.bool_) + series = pd.Series( + pd.arrays.BooleanArray(values, mask, copy=False), + copy=False, + ) else: raise ValueError( f"Frame checkpoint {path} has unknown encoding {encoding!r} for {label!r}." @@ -574,7 +663,7 @@ def _read_series( def _declared_dtype( spec: Mapping[str, Any], *, path: Path, label: str -) -> str | pd.StringDtype: +) -> str | pd.BooleanDtype | pd.StringDtype: """The concrete dtype a spec restores to, environment-independently. String dtypes resolve through the recorded storage and NA marker — the @@ -586,6 +675,8 @@ def _declared_dtype( verified in produced, and it is the build's canonical string policy. """ dtype = _require_string(spec, "dtype", label=label) + if dtype == "boolean": + return pd.BooleanDtype() if dtype not in ("str", "string"): return dtype storage = spec.get("string_storage", "python") @@ -605,7 +696,12 @@ def _declared_dtype( ) -def _dtype_matches(actual: Any, expected: str | pd.StringDtype) -> bool: +def _dtype_matches( + actual: Any, + expected: str | pd.BooleanDtype | pd.StringDtype, +) -> bool: + if isinstance(expected, pd.BooleanDtype): + return actual == expected if isinstance(expected, pd.StringDtype): return actual == expected and getattr(actual, "storage", None) == ( expected.storage @@ -737,14 +833,69 @@ def _read_metadata(root: Any, path: Path) -> dict[str, Any]: f"{metadata.get('artifact_kind')!r}." ) version = metadata.get("schema_version") - if version != FRAME_CHECKPOINT_SCHEMA_VERSION: + if version not in _SUPPORTED_FRAME_CHECKPOINT_SCHEMA_VERSIONS: raise ValueError( f"Frame checkpoint {path} schema version is {version!r}; expected " - f"{FRAME_CHECKPOINT_SCHEMA_VERSION}." + f"one of {sorted(_SUPPORTED_FRAME_CHECKPOINT_SCHEMA_VERSIONS)}." + ) + nullable_spec_count = 0 + for label, spec in _checkpoint_series_specs(metadata): + dtype = spec.get("dtype") + encoding = spec.get("encoding") + declares_nullable = dtype == "boolean" + uses_nullable_encoding = encoding == _ENCODING_NULLABLE_BOOLEAN + if version == _LEGACY_FRAME_CHECKPOINT_SCHEMA_VERSION and ( + declares_nullable or uses_nullable_encoding + ): + raise ValueError( + f"Frame checkpoint {path} schema version 2 cannot carry nullable " + f"boolean spec {label!r}." + ) + if version == FRAME_CHECKPOINT_SCHEMA_VERSION and ( + declares_nullable != uses_nullable_encoding + ): + raise ValueError( + f"Frame checkpoint {path} schema version 3 nullable boolean spec " + f"{label!r} must pair declared dtype 'boolean' with encoding " + f"{_ENCODING_NULLABLE_BOOLEAN!r}." + ) + if uses_nullable_encoding: + nullable_spec_count += 1 + if type(spec.get("has_null_mask")) is not bool: + raise ValueError( + f"Frame checkpoint {path} {label!r} nullable boolean " + "has_null_mask must be a boolean." + ) + if version == FRAME_CHECKPOINT_SCHEMA_VERSION and nullable_spec_count == 0: + raise ValueError( + f"Frame checkpoint {path} schema version 3 requires at least one " + "nullable boolean spec." ) return metadata +def _checkpoint_series_specs( + metadata: Mapping[str, Any], +) -> tuple[tuple[str, dict[str, Any]], ...]: + """Return every serialized series spec for version-policy validation.""" + + specs: list[tuple[str, dict[str, Any]]] = [] + for table_index, raw_table in enumerate(_require_list(metadata, "tables")): + table = _require_dict(raw_table, f"tables[{table_index}]") + name = table.get("name", f"tables[{table_index}]") + index = _require_dict(table.get("index"), f"tables[{table_index}].index") + if index.get("kind") == "values": + specs.append((f"{name}.index", index)) + for column_index, raw_column in enumerate(_require_list(table, "columns")): + column = _require_dict( + raw_column, + f"tables[{table_index}].columns[{column_index}]", + ) + specs.append((f"{name}.{column.get('name', column_index)}", column)) + specs.append(("strata", _require_dict(metadata.get("strata"), "strata"))) + return tuple(specs) + + def _schema_from_metadata(metadata: Mapping[str, Any]) -> EntitySchema: raw_schema = _require_dict(metadata.get("schema"), "schema") person_entity = _require_string(raw_schema, "person_entity", label="schema") diff --git a/packages/microcosm-build/src/microcosm/build/frame_serializer_registry.py b/packages/microcosm-build/src/microcosm/build/frame_serializer_registry.py new file mode 100644 index 00000000..b903cb5a --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/frame_serializer_registry.py @@ -0,0 +1,218 @@ +"""Inventory of production serializers that can encounter Frame dtypes. + +This registry is deliberately code, rather than review prose. Its test scans +every production Python source for writable HDF handles and fails when a new +physical sink has not been classified. A serializer entry denotes a logical +Frame/table-collection sink; an exclusion denotes an HDF mutation that cannot +receive a Frame column (raw arrays or root attributes only). +""" + +from __future__ import annotations + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class HdfWriteSite: + """One function that opens an HDF file for mutation.""" + + path: str + function: str + + @property + def key(self) -> str: + return f"{self.path}::{self.function}" + + +@dataclass(frozen=True) +class FrameSerializerSpec: + """A physical serializer whose input includes one or more Frame tables.""" + + serializer_id: str + writer: HdfWriteSite + backend: str + routes: tuple[str, ...] + version_owner: str + nullable_boolean_storage: str + direct_hdf_open: bool = True + + +@dataclass(frozen=True) +class HdfWriteExclusion: + """A writable HDF site that provably does not serialize Frame columns.""" + + exclusion_id: str + writer: HdfWriteSite + reason: str + + +FRAME_TABLE_SERIALIZERS = ( + FrameSerializerSpec( + serializer_id="frame_checkpoint", + writer=HdfWriteSite( + "packages/microcosm-build/src/microcosm/build/frame_checkpoint.py", + "write_frame_checkpoint", + ), + backend="h5py", + routes=("generic Frame checkpoints", "US pool stage checkpoints"), + version_owner="FRAME_CHECKPOINT_SCHEMA_VERSION", + nullable_boolean_storage="bool_values_optional_uint8_mask", + ), + FrameSerializerSpec( + serializer_id="nullable_us_h5", + writer=HdfWriteSite( + "packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py", + "_write_nullable_us_h5_file", + ), + backend="pandas.HDFStore fixed", + routes=( + "stacked terminal pool publication", + "current legacy two-spine publication facade", + "US ACS calibrated release", + "US L0 refit export", + ), + version_owner="US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION", + nullable_boolean_storage="numpy_bool_or_object_pd_na_v1", + ), + FrameSerializerSpec( + serializer_id="uk_single_year_h5", + writer=HdfWriteSite( + "packages/microcosm-build/src/microcosm/build/uk_runtime/national_build.py", + "_write_uk_single_year_tables", + ), + backend="pandas.HDFStore table", + routes=( + "UK national publication", + "UK rowwise publication", + "UK ladder-rowwise publication", + ), + version_owner="UK single-year payload contract", + nullable_boolean_storage="numpy_bool_or_object_pd_na_v1", + ), + FrameSerializerSpec( + serializer_id="axiom_entity_tables", + writer=HdfWriteSite( + "packages/microcosm-frame/src/microcosm/frame/adapters/axiom.py", + "save", + ), + backend="pandas.HDFStore table", + routes=("Axiom adapter entity-table dataset",), + version_owner="AxiomEntityTableDataset payload contract", + nullable_boolean_storage="numpy_bool_or_object_pd_na_v1", + ), + FrameSerializerSpec( + serializer_id="policyengine_us_single_year", + writer=HdfWriteSite( + "packages/microcosm-frame/src/microcosm/frame/adapters/policyengine_us.py", + "_write_and_verify", + ), + backend="pandas.HDFStore table", + routes=("PolicyEngine-US adapter export",), + version_owner="PolicyEngineUSAdapter payload contract", + nullable_boolean_storage="numpy_bool_or_object_pd_na_v1", + ), + FrameSerializerSpec( + serializer_id="legacy_us_two_spine", + writer=HdfWriteSite( + "tools/_legacy/build_us_acs_multispine_base.py", + "_write_dataset", + ), + backend="pandas.HDFStore fixed", + routes=("preserved directly executable legacy two-spine builder",), + version_owner="legacy schema-4 publication contract", + nullable_boolean_storage="numpy_bool_or_object_pd_na_v1", + ), + FrameSerializerSpec( + serializer_id="acs_local_lean_checkpoint", + writer=HdfWriteSite( + "tools/build_us_acs_local_release.py", + "write_lean_checkpoint", + ), + backend="pandas.HDFStore fixed", + routes=("US ACS local lean target-frame checkpoint",), + version_owner="ACS local checkpoint payload contract", + nullable_boolean_storage="numpy_bool_or_object_pd_na_v1", + ), + FrameSerializerSpec( + serializer_id="fiscal_target_frame_checkpoint", + writer=HdfWriteSite( + "tools/build_us_fiscal_refresh_release.py", + "_write_target_frame_checkpoint", + ), + backend="h5py", + routes=("US fiscal-refresh target-frame checkpoint",), + version_owner="TARGET_FRAME_CHECKPOINT_SCHEMA_VERSION", + nullable_boolean_storage="bool_values_optional_uint8_mask", + ), +) + + +HDF_WRITE_EXCLUSIONS = ( + HdfWriteExclusion( + exclusion_id="l0_refit_root_attrs", + writer=HdfWriteSite( + "packages/microcosm-build/src/microcosm/build/us_runtime/" + "l0_refit_export.py", + "copy_microcosm_root_attrs", + ), + reason="Copies Microcosm-owned root attributes only.", + ), + HdfWriteExclusion( + exclusion_id="acs_transfer_raw_draw_bank", + writer=HdfWriteSite( + "packages/microcosm-build/src/microcosm/build/us_runtime/" + "acs_transfer_bank.py", + "write_target", + ), + reason="Writes canonical JSON bytes and raw numeric draw bits only.", + ), + HdfWriteExclusion( + exclusion_id="puf_qrf_raw_draw_checkpoint", + writer=HdfWriteSite( + "packages/microcosm-build/src/microcosm/build/us_runtime/puf_qrf_chain.py", + "_write_target_checkpoint", + ), + reason="Writes canonical JSON bytes and raw numeric draw bits only.", + ), + HdfWriteExclusion( + exclusion_id="uk_weight_root_attrs", + writer=HdfWriteSite( + "packages/microcosm-build/src/microcosm/build/uk_runtime/national_build.py", + "_write_weight_metadata", + ), + reason="Adds weight-kind and mass-log root attributes only.", + ), + HdfWriteExclusion( + exclusion_id="puf_equivalence_raw_draw_observer", + writer=HdfWriteSite( + "tools/build_us_puf_support_base.py", + "observe_primary_qrf", + ), + reason="Optional equivalence observer writes float draw bits and attrs only.", + ), + HdfWriteExclusion( + exclusion_id="puf_monolith_geography_attrs", + writer=HdfWriteSite( + "tools/build_us_puf_support_base.py", + "_run_all", + ), + reason="Adds geography provenance root attributes only.", + ), + HdfWriteExclusion( + exclusion_id="puf_staged_geography_attrs", + writer=HdfWriteSite( + "tools/build_us_puf_support_base.py", + "_export_staged_result", + ), + reason="Adds geography provenance root attributes only.", + ), +) + + +__all__ = [ + "FRAME_TABLE_SERIALIZERS", + "HDF_WRITE_EXCLUSIONS", + "FrameSerializerSpec", + "HdfWriteExclusion", + "HdfWriteSite", +] diff --git a/packages/microcosm-build/src/microcosm/build/logbook.py b/packages/microcosm-build/src/microcosm/build/logbook.py index 8ef72df7..3312ae06 100644 --- a/packages/microcosm-build/src/microcosm/build/logbook.py +++ b/packages/microcosm-build/src/microcosm/build/logbook.py @@ -90,7 +90,7 @@ ) _DIGEST_PATTERN = re.compile(r"^[0-9a-f]{64}$") _BUILD_ID_PATTERN = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,254}$") -LOGBOOK_RUNGS = frozenset({"f001", "f010", "f100"}) +LOGBOOK_RUNGS = frozenset({"f001", "f004", "f010", "f100"}) LEDGER_API_KEY_ENV = "POPULACE_LEDGER_API_KEY" LOGBOOK_ROW_FIELDS = frozenset( { @@ -964,7 +964,7 @@ def _normalize_timestamp(value: str | datetime, field: str) -> str: def _validate_rung(value: str) -> str: if not isinstance(value, str) or value not in LOGBOOK_RUNGS: raise ValueError( - "rung must be a #624 fraction token: 'f001', 'f010', or 'f100'." + "rung must be a #624 fraction token: 'f001', 'f004', 'f010', or 'f100'." ) return value diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/national_build.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/national_build.py index 62b5cf93..fd491949 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/national_build.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/national_build.py @@ -66,7 +66,14 @@ UKReviewedExclusion, ) from microcosm.build.uk_runtime.weighted_integrity import exclusion_evaluation_date -from microcosm.frame import Frame, MassChangeRecord, WeightKind, engine_tables +from microcosm.frame import ( + Frame, + MassChangeRecord, + WeightKind, + engine_tables, + put_frame_table, + read_frame_table, +) __all__ = [ "UKNationalBuildResult", @@ -191,9 +198,9 @@ def _read_uk_national_tables( "UK national dataset time_period must contain exactly one value." ) payload = { - "person": store["person"], - "benunit": store["benunit"], - "household": store["household"], + "person": read_frame_table(store, "person"), + "benunit": read_frame_table(store, "benunit"), + "household": read_frame_table(store, "household"), "time_period": str(raw_period.iloc[0]), "household_weight_kind": _weight_kind_from_stored(stored_kind), "mass_log": _mass_log_from_stored(stored_mass_log), @@ -254,9 +261,27 @@ def _write_uk_single_year_tables( temporary_path = path.with_name(f".{path.name}.{uuid.uuid4().hex}.tmp.h5") try: with pd.HDFStore(temporary_path) as store: - store.put("person", person, format="table", data_columns=True) - store.put("benunit", benunit, format="table", data_columns=True) - store.put("household", household, format="table", data_columns=True) + put_frame_table( + store, + "person", + person, + preferred_format="table", + data_columns=True, + ) + put_frame_table( + store, + "benunit", + benunit, + preferred_format="table", + data_columns=True, + ) + put_frame_table( + store, + "household", + household, + preferred_format="table", + data_columns=True, + ) store.put( "time_period", pd.Series([time_period]), diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py index 53f94f40..e55690cb 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py @@ -216,6 +216,7 @@ US_AOTC_ELIGIBILITY_OUTPUT_COLUMNS, US_EDUCATION_INPUTS_NONCONSTANT_PERSON_COLUMNS, US_EDUCATION_INPUTS_OUTPUT_COLUMNS, + US_EDUCATION_INPUTS_OWNED_OUTPUT_COLUMNS, US_EDUCATION_INPUTS_REQUIRED_SOURCE_COLUMNS, US_EDUCATION_INPUTS_STAGE_NAME, derive_us_education_inputs_from_manifest, @@ -540,13 +541,17 @@ PUF_CAPITAL_GAINS_TAIL_QUANTILE, PUF_CAPITAL_GAINS_TAIL_STAGE_NAME, PUF_CAPITAL_GAINS_TAIL_SUPPORT_CHANNEL, + PUF_CAPITAL_GAINS_TAIL_SUPPORT_CONTRACT_VERSION, PUF_CAPITAL_GAINS_TAIL_TAX_UNIT_COLUMNS, PUF_CAPITAL_GAINS_TAIL_TRANSFER_WEIGHT_COLUMN, assert_puf_capital_gains_tail_survives_selection, puf_capital_gains_tail_concentration_gate, + puf_capital_gains_tail_support_contract_identity, + puf_capital_gains_tail_terminal_support_receipt, select_puf_capital_gains_tail_donors, transfer_puf_capital_gains_tail, validate_puf_capital_gains_tail_manifest, + validate_puf_capital_gains_tail_terminal_support_receipt, write_puf_capital_gains_tail_manifest, ) from microcosm.build.us_runtime.puf_donor_io import load_puf_tax_unit_donor @@ -1448,6 +1453,7 @@ "US_AOTC_ELIGIBILITY_OUTPUT_COLUMNS", "US_EDUCATION_INPUTS_NONCONSTANT_PERSON_COLUMNS", "US_EDUCATION_INPUTS_OUTPUT_COLUMNS", + "US_EDUCATION_INPUTS_OWNED_OUTPUT_COLUMNS", "ASEC_EDUCATION_ASSISTANCE_ARCHIVES", "ASEC_EDUCATION_ASSISTANCE_INCOME_YEARS", "fetch_asec_education_assistance_source", @@ -1811,6 +1817,7 @@ "PUF_CAPITAL_GAINS_TAIL_POSITIVE_MASS_FIVE_X_TARGET", "PUF_CAPITAL_GAINS_TAIL_QUANTILE", "PUF_CAPITAL_GAINS_TAIL_STAGE_NAME", + "PUF_CAPITAL_GAINS_TAIL_SUPPORT_CONTRACT_VERSION", "PUF_CAPITAL_GAINS_TAIL_SUPPORT_CHANNEL", "PUF_CAPITAL_GAINS_TAIL_TAX_UNIT_COLUMNS", "PUF_CAPITAL_GAINS_TAIL_TRANSFER_WEIGHT_COLUMN", @@ -1865,6 +1872,8 @@ "load_asec_h5_tables", "out_of_sample_reform_specs", "puf_capital_gains_tail_concentration_gate", + "puf_capital_gains_tail_support_contract_identity", + "puf_capital_gains_tail_terminal_support_receipt", "puf_tax_unit_donor_from_arrays", "pool_asec_sources", "prepare_us_puf_tax_detail_chain_inputs", @@ -1950,6 +1959,7 @@ "puf_processed_capital_gains_stage", "puf_raw_e01000_stage", "validate_puf_capital_gains_tail_manifest", + "validate_puf_capital_gains_tail_terminal_support_receipt", "validation_only_family_ids", "translate_congressional_district_facts_to_current_vintage", "with_household_congressional_districts", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_income_universe.py b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_income_universe.py index 44b5730d..8214f07c 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_income_universe.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_income_universe.py @@ -29,6 +29,7 @@ "ACS_PUMS_2024_DATA_DICTIONARY_URL", "ACS_PUMS_EARNINGS_MINIMUM_AGE", "ACS_PUMS_EARNINGS_SOURCE_COLUMNS", + "ACS_PUMS_EARNINGS_UNIVERSE_PERSON_INPUTS", "AcsPumsEarningsUniverse", "AcsPumsEarningsUniverseApplication", "acs_pums_earnings_universe_contract_identity", @@ -48,6 +49,19 @@ } ) +# Physical person-table inputs owned by this provenance-aware universe +# resolver. Population-treatment modules consume this declaration without +# learning or constructing source-channel column names themselves. +ACS_PUMS_EARNINGS_UNIVERSE_PERSON_INPUTS: Mapping[str, str] = MappingProxyType( + { + "age": "assembled_native_person_input", + "person_tax_unit_id": "frame_membership", + support_channel_column("person"): "assembly_support_provenance", + support_clone_index_column("person"): "assembly_support_provenance", + support_source_id_column("person"): "assembly_support_source_identity", + } +) + _ACS_SUPPORT_CHANNEL = "acs" _RULE_VERSION = 1 _UNIVERSE_DESCRIPTION = "ACS persons age 15 and older" @@ -324,7 +338,7 @@ def resolve_acs_pums_earnings_universe( person, column=column, raw_source_column=source_column, - scope=scope, + scope=acs_scope, ) if raw_source_present else None diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py index 8034fa85..09e2c90e 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py @@ -19,6 +19,7 @@ from __future__ import annotations import hashlib +import json from collections.abc import Iterable, Mapping, Sequence from dataclasses import dataclass, replace from functools import lru_cache @@ -30,6 +31,9 @@ from microcosm.build.gates import FitWeightRecord from microcosm.build.serialization_dtypes import canonicalize_frame_string_dtypes +from microcosm.build.us_runtime.capital_gain_distributions import ( + capital_gain_distribution_shares_asset_identity, +) from microcosm.build.us_runtime.puf_support import ( PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS, PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS, @@ -62,6 +66,7 @@ "AcsTransferTargetCheckpoint", "TargetFamilies", "acs_transfer_donor_requirements", + "acs_transfer_execution_contract_identity", "assert_acs_transfer_targets_are_input_leaves", "declared_acs_transfer_target_families", "default_acs_transfer_target_families", @@ -219,8 +224,81 @@ "RENTED": 3.0, "RENTER": 3.0, } -_ACS_TENURE_CODES: Mapping[int, float] = {1: 1.0, 2: 2.0, 3: 3.0, 4: 0.0} -_CPS_TENURE_CODES: Mapping[int, float] = {1: 1.0, 2: 3.0, 3: 0.0} + + +def acs_transfer_execution_contract_identity( + *, + targets: Sequence[str] | None = None, + derive_schedule_d: bool = True, +) -> dict[str, object]: + """Bind the complete runtime predictor and target-codec contract.""" + + requested_targets = frozenset(() if targets is None else targets) + schedule_d_enabled = derive_schedule_d and bool( + requested_targets & {_SCHEDULE_D_CGD_SOURCE, _SCHEDULE_D_CGD_EXCLUSIVE_WITH} + ) + adult_care_enabled = _ADULT_CARE_EXPENSE in requested_targets + payload: dict[str, object] = { + "schema_version": 2, + "person_required_predictors": list(ACS_PERSON_TRANSFER_PREDICTORS), + "person_optional_predictors": list(ACS_OPTIONAL_PERSON_TRANSFER_PREDICTORS), + "group_required_predictors": list(ACS_GROUP_TRANSFER_PREDICTORS), + "group_optional_names": dict(sorted(_GROUP_OPTIONAL_NAMES.items())), + "donor_combined_components": { + feature: list(columns) + for feature, columns in sorted(_DONOR_COMBINED_COMPONENTS.items()) + }, + "recipient_combined_sources": dict(sorted(_RECIPIENT_COMBINED_SOURCES.items())), + "housing": { + "targets": sorted(_HOUSING_TRANSFER_TARGETS), + "mandatory_features": [_HEAD_FEATURE, _TENURE_FEATURE], + "head_source_precedence": [ + {"source": "is_household_head", "head_codes": [True]}, + {"source": "A_EXPRRP", "head_codes": [1, 2]}, + {"source": "A_LINENO", "head_codes": [1]}, + ], + "tenure_source_precedence": [ + "tenure_type", + "spm_unit_tenure_type", + ], + }, + "tenure_codes": dict(sorted(_TENURE_CODES.items())), + "immigration_status_targets": list(_IMMIGRATION_STATUS_TARGETS), + "immigration_status_model_target": _IMMIGRATION_STATUS_MODEL_TARGET, + "discrete_numeric_targets": sorted(_DISCRETE_NUMERIC_TARGETS), + "post_transfer_structure": { + "schedule_d_capital_gain_distributions": { + "enabled": schedule_d_enabled, + "source": _SCHEDULE_D_CGD_SOURCE, + "exclusive_with": _SCHEDULE_D_CGD_EXCLUSIVE_WITH, + "output": _SCHEDULE_D_CGD_COLUMN, + "preserve_preexisting_nonnull": True, + "share_asset": ( + capital_gain_distribution_shares_asset_identity() + if schedule_d_enabled + else None + ), + }, + "adult_care": { + "enabled": adult_care_enabled, + "flag": _ADULT_CARE_FLAG, + "expense": _ADULT_CARE_EXPENSE, + "tax_unit_role": _ADULT_CARE_ROLE, + "tax_unit_link": _ADULT_CARE_UNIT, + "mutable_rows": "newly_imputed_expense_cells_only", + }, + }, + } + payload["sha256"] = hashlib.sha256( + json.dumps( + payload, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + ).hexdigest() + return payload + type TargetFamilies = Mapping[str, Mapping[str, Sequence[str]]] @@ -526,6 +604,8 @@ def acs_derived_transfer_expectations( def derive_acs_schedule_d_capital_gain_distributions( person: pd.DataFrame, + *, + share: float | None = None, ) -> tuple[np.ndarray, dict[str, object]]: """Re-derive the Schedule D CGD memo leg from transferred parents. @@ -540,8 +620,17 @@ def derive_acs_schedule_d_capital_gain_distributions( load_capital_gain_distribution_shares, ) - share = load_capital_gain_distribution_shares() - ratio = float(share.schedule_d_cgd_share_of_lt_net_gains) + ratio = ( + float( + load_capital_gain_distribution_shares().schedule_d_cgd_share_of_lt_net_gains + ) + if share is None + else float(share) + ) + if not np.isfinite(ratio) or not 0.0 < ratio < 1.0: + raise ValueError( + "Schedule D capital-gain-distribution share must be finite in (0, 1)." + ) source = pd.to_numeric(person[_SCHEDULE_D_CGD_SOURCE], errors="coerce") other_route = pd.to_numeric(person[_SCHEDULE_D_CGD_EXCLUSIVE_WITH], errors="coerce") if source.isna().any() or other_route.isna().any(): @@ -704,7 +793,6 @@ def acs_transfer_donor_requirements( source for source in ( "is_household_head", - "RELSHIPP", "A_EXPRRP", "A_LINENO", ) @@ -725,8 +813,6 @@ def acs_transfer_donor_requirements( for source in ( "tenure_type", "spm_unit_tenure_type", - "TEN", - "H_TENURE", ) if (owner := _column_owner_or_none(donor, source)) is not None ), @@ -802,6 +888,8 @@ def transfer_acs_inputs( n_estimators: int = 100, max_targets_per_fit: int = DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, target_bank: AcsTransferTargetBank | None = None, + derive_schedule_d: bool = True, + execution_contract: Mapping[str, object] | None = None, ) -> AcsTransferResult: """Impute requested missing leaves from ``donor`` onto ``recipient``. @@ -864,6 +952,16 @@ def transfer_acs_inputs( all_targets = [ target for _entity, _family, targets in requested for target in targets ] + resolved_execution_contract = acs_transfer_execution_contract_identity( + targets=all_targets, + derive_schedule_d=derive_schedule_d, + ) + if execution_contract is not None and dict(execution_contract) != ( + resolved_execution_contract + ): + raise ValueError( + "ACS transfer runtime execution contract differs from its bound input." + ) assert_acs_transfer_targets_are_input_leaves(all_targets) active = _missing_target_families(requested, recipient=recipient) @@ -969,6 +1067,7 @@ def transfer_acs_inputs( imputed_masks=imputed_masks, donor_spine=donor_spine, resolved_channel=resolved_channel, + execution_contract=resolved_execution_contract, ) tables: dict[str, pd.DataFrame] = dict(output_tables) @@ -1005,6 +1104,7 @@ def _apply_post_transfer_structure( imputed_masks: Mapping[tuple[str, str], np.ndarray], donor_spine: str, resolved_channel: str | None, + execution_contract: Mapping[str, object], ) -> None: """Apply the deterministic post-fit steps the base's construction implies. @@ -1021,6 +1121,12 @@ def _apply_post_transfer_structure( if person is None: return + post_transfer_contract = execution_contract["post_transfer_structure"] + assert isinstance(post_transfer_contract, Mapping) + schedule_d_contract = post_transfer_contract[ + "schedule_d_capital_gain_distributions" + ] + assert isinstance(schedule_d_contract, Mapping) cgd_parents = {_SCHEDULE_D_CGD_SOURCE, _SCHEDULE_D_CGD_EXCLUSIVE_WITH} cgd_candidate = np.zeros(len(person), dtype=bool) for parent in cgd_parents: @@ -1028,7 +1134,11 @@ def _apply_post_transfer_structure( ("person", parent), np.zeros(len(person), dtype=bool), ) - if cgd_candidate.any() and cgd_parents <= set(person.columns): + if ( + schedule_d_contract["enabled"] is True + and cgd_candidate.any() + and cgd_parents <= set(person.columns) + ): source = pd.to_numeric(person[_SCHEDULE_D_CGD_SOURCE], errors="coerce") other_route = pd.to_numeric( person[_SCHEDULE_D_CGD_EXCLUSIVE_WITH], @@ -1052,7 +1162,12 @@ def _apply_post_transfer_structure( derivation: dict[str, object] = {} else: values, derivation = derive_acs_schedule_d_capital_gain_distributions( - person.loc[fill] + person.loc[fill], + share=float( + schedule_d_contract["share_asset"][ + "schedule_d_cgd_share_of_lt_net_gains" + ] + ), ) derived_output.loc[fill] = values person[_SCHEDULE_D_CGD_COLUMN] = derived_output @@ -1861,7 +1976,6 @@ def _person_head_feature(frame: Frame) -> pd.Series | None: return direct.rename(_HEAD_FEATURE) for source, head_codes in ( - ("RELSHIPP", {20}), ("A_EXPRRP", {1, 2}), ("A_LINENO", {1}), ): @@ -1902,30 +2016,6 @@ def _person_tenure_feature(frame: Frame) -> pd.Series | None: ) return pd.Series(mapped, index=frame.person.index, name=_TENURE_FEATURE) - for source, codes in ( - ("TEN", _ACS_TENURE_CODES), - ("H_TENURE", _CPS_TENURE_CODES), - ): - values = _column_broadcast_to_person(frame, source) - if values is None: - continue - raw = _numeric_source(values, context=f"tenure predictor {source}") - mapped = np.full(len(raw), np.nan, dtype=np.float64) - invalid: list[float] = [] - for position, value in enumerate(raw): - if np.isnan(value): - continue - code = int(value) - if value != code or code not in codes: - invalid.append(float(value)) - continue - mapped[position] = codes[code] - if invalid: - raise ValueError( - f"ACS transfer tenure predictor {source!r} contains unsupported " - f"code(s): {invalid[:5]}." - ) - return pd.Series(mapped, index=frame.person.index, name=_TENURE_FEATURE) return None @@ -2690,11 +2780,12 @@ def _target_encoding(series: pd.Series, *, target: str) -> _TargetEncoding: ) if target_dtype == "bool" or semantic_boolean: - # Primary PUF finalization deliberately stores its QBI boolean-count - # outputs in physical float columns before they become ACS-transfer - # donors (including through the supported legacy HDF path). Keep that - # numeric-dtype 0/1 compatibility explicit; object-backed 0/1 values - # are rejected above rather than coerced through metadata. + # Retiring primary-PUF and HDF artifacts can carry boolean targets in + # the audited physical numeric 0/1 representation. Keep that legacy + # compatibility explicit; the canonical stacked late path now + # materializes booleans physically and validates every callback output. + # Object-backed 0/1 values remain rejected rather than being coerced + # through metadata. values = pd.Series( _as_float_array(series), index=series.index, diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/capital_gain_distributions.py b/packages/microcosm-build/src/microcosm/build/us_runtime/capital_gain_distributions.py index d0382eac..2a7be76b 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/capital_gain_distributions.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/capital_gain_distributions.py @@ -20,6 +20,7 @@ from __future__ import annotations +import hashlib import json from collections.abc import Mapping from dataclasses import dataclass @@ -37,6 +38,7 @@ __all__ = [ "CapitalGainDistributionShares", + "capital_gain_distribution_shares_asset_identity", "load_capital_gain_distribution_shares", "split_us_component_by_share_from_manifest", ] @@ -119,9 +121,31 @@ def load_capital_gain_distribution_shares() -> CapitalGainDistributionShares: """Load and validate the packaged SOCA-derived share resource.""" path = files("microcosm.build.us") / f"{_SHARES_RESOURCE_NAME}.json" - return CapitalGainDistributionShares.from_dict( - json.loads(path.read_text(encoding="utf-8")) - ) + return _capital_gain_distribution_shares_from_bytes(path.read_bytes()) + + +def _capital_gain_distribution_shares_from_bytes( + payload: bytes, +) -> CapitalGainDistributionShares: + """Resolve validated share semantics from the exact bytes being bound.""" + + return CapitalGainDistributionShares.from_dict(json.loads(payload.decode("utf-8"))) + + +def capital_gain_distribution_shares_asset_identity() -> dict[str, object]: + """Bind the exact packaged bytes and resolved Schedule-D share semantics.""" + + path = files("microcosm.build.us") / f"{_SHARES_RESOURCE_NAME}.json" + payload = path.read_bytes() + resolved = _capital_gain_distribution_shares_from_bytes(payload) + return { + "asset": f"microcosm.build.us/{_SHARES_RESOURCE_NAME}.json", + "asset_sha256": hashlib.sha256(payload).hexdigest(), + "schedule_d_cgd_share_of_lt_net_gains": ( + resolved.schedule_d_cgd_share_of_lt_net_gains + ), + "national_anchor_ty2015": dict(resolved.anchor), + } def split_us_component_by_share_from_manifest( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/education_inputs.py b/packages/microcosm-build/src/microcosm/build/us_runtime/education_inputs.py index 0d7829d6..6ec8137f 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/education_inputs.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/education_inputs.py @@ -50,6 +50,7 @@ __all__ = [ "US_AOTC_ELIGIBILITY_OUTPUT_COLUMNS", "US_EDUCATION_INPUTS_NONCONSTANT_PERSON_COLUMNS", + "US_EDUCATION_INPUTS_OWNED_OUTPUT_COLUMNS", "US_EDUCATION_INPUTS_OUTPUT_COLUMNS", "US_EDUCATION_INPUTS_REQUIRED_SOURCE_COLUMNS", "US_EDUCATION_INPUTS_STAGE_NAME", @@ -76,6 +77,13 @@ *US_AOTC_ELIGIBILITY_OUTPUT_COLUMNS, ) +# Qualified tuition is produced upstream by the PUF tax-detail operator. This +# stage validates and consumes it, but must not claim or rewrite it. +US_EDUCATION_INPUTS_OWNED_OUTPUT_COLUMNS: tuple[str, ...] = ( + "educational_assistance", + *US_AOTC_ELIGIBILITY_OUTPUT_COLUMNS, +) + US_EDUCATION_INPUTS_NONCONSTANT_PERSON_COLUMNS = US_EDUCATION_INPUTS_OUTPUT_COLUMNS US_EDUCATION_INPUTS_REQUIRED_SOURCE_COLUMNS: tuple[str, ...] = ( @@ -84,18 +92,10 @@ ) _PERSON_WEIGHT_COLUMN = "person_weight" -# Qualified tuition is a PUF-only tax-detail chain target: on the two-channel -# support pool it is exactly zero on the ASEC half BY DESIGN (like every other -# PUF-only detail column — partnership income, charitable donations, mortgage -# interest all decompose the same way on the full-scale pool), and the PUF -# half carries the PUF-faithful per-person rate (~0.56% weighted on the first -# full-scale base, base-r3 ck016). The retired extended-CPS 2.64% person rate -# is a blended post-imputation file rate and is not comparable to this -# pre-selection pool, which is how the original [0.003, 0.05] total floor was -# mis-set. The gate now asserts the channel invariant directly: ASEC exactly -# zero, PUF share within band, blended total within band. +# Qualified tuition is a PUF-owned tax-detail input. The late producer DAG +# completes its recipient rows before this stage derives the five AOTC facts, +# so the terminal surface is intentionally no longer channel-sparse. _TUITION_SHARE_BAND = (0.001, 0.05) -_TUITION_PUF_CHANNEL_SHARE_BAND = (0.002, 0.05) _ASSISTANCE_SHARE_BAND = (0.005, 0.08) _DERIVE_EDUCATION_INPUTS_PARAMETER_KEYS = frozenset() @@ -126,7 +126,7 @@ def derive_us_education_inputs_from_manifest( operation: SourceOperationSpec, _context: SourceRuntimeContext | None, ) -> pd.DataFrame: - """Derive the seven education inputs from a person source table.""" + """Derive assistance and AOTC flags while preserving upstream tuition.""" if operation.kind != "derive_education_inputs": raise SourceRuntimeError( @@ -155,20 +155,31 @@ def derive_us_education_inputs_from_manifest( f"US education-input derivation requires source column(s): {missing}." ) + def _strict_nonnegative(column: str) -> np.ndarray: + values = pd.to_numeric(frame[column], errors="coerce").to_numpy( + dtype=np.float64 + ) + nonfinite = int(np.count_nonzero(~np.isfinite(values))) + if nonfinite: + raise SourceRuntimeError( + f"US education-input source {column!r} contains {nonfinite} " + "nonnumeric or nonfinite value(s)." + ) + negative = int(np.count_nonzero(values < 0.0)) + if negative: + raise SourceRuntimeError( + f"US education-input source {column!r} contains {negative} " + "negative value(s)." + ) + return values + + tuition = _strict_nonnegative("qualified_tuition_expenses") + assistance = _strict_nonnegative("ED_VAL") result = frame.copy(deep=True) - tuition = ( - pd.to_numeric(result["qualified_tuition_expenses"], errors="coerce") - .fillna(0.0) - .clip(lower=0.0) - ) - assistance = ( - pd.to_numeric(result["ED_VAL"], errors="coerce").fillna(0.0).clip(lower=0.0) - ) aotc_student = tuition > 0.0 - result["qualified_tuition_expenses"] = tuition.to_numpy(dtype=np.float64) - result["educational_assistance"] = assistance.to_numpy(dtype=np.float64) + result["educational_assistance"] = assistance for column in US_AOTC_ELIGIBILITY_OUTPUT_COLUMNS: - result[column] = aotc_student.to_numpy(dtype=bool) + result[column] = aotc_student return result @@ -215,7 +226,7 @@ def with_us_education_inputs( config=SourceRuntimeConfig(seed=int(seed), target_year=int(time_period)), ) aligned = output.set_index("person_id").reindex(person["person_id"]) - for column in US_EDUCATION_INPUTS_OUTPUT_COLUMNS: + for column in US_EDUCATION_INPUTS_OWNED_OUTPUT_COLUMNS: if aligned[column].isna().any(): raise ValueError( "US education-input stage output does not cover every person for " @@ -223,7 +234,7 @@ def with_us_education_inputs( ) tables = {entity: frame.table(entity).copy() for entity in frame.entities} - for column in US_EDUCATION_INPUTS_OUTPUT_COLUMNS: + for column in US_EDUCATION_INPUTS_OWNED_OUTPUT_COLUMNS: if column in US_AOTC_ELIGIBILITY_OUTPUT_COLUMNS: tables["person"][column] = aligned[column].to_numpy(dtype=bool) else: @@ -353,23 +364,6 @@ def us_education_inputs_signal_gate(frame: Frame) -> GateResult: f"{column}: {count} rows disagree with positive qualified tuition." ) - channels = summary.get("channels") or {} - if channels: - asec = channels.get(BASE_ASEC_SUPPORT_CHANNEL, {}) - if int(asec.get("tuition_positive_rows", 0)): - failures.append( - "qualified_tuition_expenses is a PUF-only tax-detail column but " - f"{asec['tuition_positive_rows']} ASEC-channel row(s) carry it; " - "the support channels have cross-contaminated." - ) - puf = channels.get(PUF_TAX_DETAIL_SUPPORT_CHANNEL, {}) - puf_share = float(puf.get("tuition_positive_share", 0.0)) - puf_low, puf_high = _TUITION_PUF_CHANNEL_SHARE_BAND - if not (puf_low <= puf_share <= puf_high): - failures.append( - f"puf_tax_detail qualified-tuition share {puf_share:.4f} outside " - f"plausibility band [{puf_low}, {puf_high}]." - ) return GateResult( name="education_inputs_signal", passed=not failures, diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py b/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py index c3085141..fcd155d4 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py @@ -15,7 +15,6 @@ import re import shutil import uuid -import warnings from collections.abc import Mapping from dataclasses import dataclass from pathlib import Path @@ -27,7 +26,14 @@ canonicalize_frame_string_dtypes, canonicalize_table_string_dtypes, ) -from microcosm.frame import Frame, WeightKind, Weights +from microcosm.frame import ( + Frame, + WeightKind, + Weights, + materialize_nullable_booleans_for_pytables, + put_frame_table, + read_frame_table, +) from microcosm.frame.units import US_SCHEMA __all__ = [ @@ -35,9 +41,11 @@ "AuthenticatedPoolH5MismatchError", "LEGACY_NULLABLE_STAGING_ARTIFACT_KIND", "US_MULTISPINE_AGREEMENT_DIAGNOSTICS_ARTIFACT_KIND", + "US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION", "US_MULTISPINE_POOL_H5_ARTIFACT_KIND", "US_MULTISPINE_POOL_MANIFEST_ARTIFACT_KIND", "US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION", + "US_STACKED_POOL_OPERATOR_ORDER", "load_legacy_calibrated_us_h5", "load_simulation_ready_us_multispine_pool", "load_simulation_ready_us_multispine_pool_manifest", @@ -51,12 +59,178 @@ US_MULTISPINE_AGREEMENT_DIAGNOSTICS_ARTIFACT_KIND = ( "populace_us_multispine_agreement_diagnostics" ) -# 4 adds identity-bound stage-checkpoint provenance and an explicit always-fresh -# terminal agreement receipt to the companion pool manifest. -US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION = 4 +# 8 binds the nullable-boolean-capable physical H5 materializer in both the +# stacked manifest receipt and the H5's frozen metadata key. +# 7 binds the complete late-producer resource semantics and removes the PUF +# callback's duplicate outer-order entry; the callback is a node inside the DAG. +# 6 additionally bound the independently carried late-producer transition +# authority and restores its immutable Frame-metadata anchor on H5 load. +# Schema 5 can authenticate the DAG receipt's structure, but cannot prove that +# the published receipt is the one authorized by the generating transition. +US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION = 8 +US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION = 2 +"""Stacked terminal H5 materializer; version 2 handles pandas BooleanDtype.""" +_LEGACY_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION = 4 _METADATA_KEY = "_populace_staging_metadata" _TIME_PERIOD_KEY = "_time_period" _LOWERCASE_SHA256 = re.compile(r"[0-9a-f]{64}") +_STACKED_PIPELINE = "us-stacked-pool" +US_STACKED_POOL_OPERATOR_ORDER = ( + "assemble_stacked_spine", + "prepare_multispine_source_inputs_for_clone", + "gap_fill_stacked_spine", + "run_stacked_late_producer_dag", + "prepare_stacked_tail_derivation", + "derive_multispine_pool_inputs", + "seed_multispine_pool_inputs", + "materialize_multispine_agreement_outputs", + "stacked_completeness_gate", + "by_origin_battery", +) +_LEGACY_POOL_OPERATOR_ORDER = ( + "assemble", + "clone", + "impute", + "derive", + "seed", + "simulate", + "agreement", +) +_LEGACY_POOL_CHECKPOINT_ARTIFACT_KIND = ( + "populace_us_multispine_pool_checkpoint_provenance" +) +_LEGACY_POOL_CHECKPOINT_SCHEMA_VERSION = 1 +_LEGACY_POOL_CHECKPOINT_MATERIALIZER_VERSION = 3 +_LEGACY_REQUIRED_STAGE_RECEIPTS = frozenset({"impute", "derive", "seed", "simulate"}) +_STACKED_ONLY_MANIFEST_FIELDS = frozenset( + { + "pipeline", + "release_id", + "sampling", + "clone_attachment", + "input_pins_digest", + "late_producer_transition_authority_sha256", + "stack_manifest", + "terminal_gates", + } +) +_REQUIRED_STACKED_MANIFEST_FIELDS = frozenset( + { + "pipeline", + "operator_order", + "stage_receipts", + } +) + + +def _stacked_manifest_markers(manifest: Mapping[str, object]) -> set[str]: + """Return every top-level or nested field that proves stacked lineage.""" + + markers = set(manifest) & set(_STACKED_ONLY_MANIFEST_FIELDS) + operator_order = manifest.get("operator_order") + if isinstance(operator_order, list) and any( + operator + in { + "assemble_stacked_spine", + "run_stacked_late_producer_dag", + "by_origin_battery", + } + for operator in operator_order + ): + markers.add("operator_order[stacked]") + stage_receipts = manifest.get("stage_receipts") + impute = ( + stage_receipts.get("impute") if isinstance(stage_receipts, Mapping) else None + ) + if isinstance(impute, Mapping) and set(impute) & { + "stacked_late_producer_dag", + "stacked_post_puf_transfer", + }: + markers.add("stage_receipts.impute[stacked]") + pool_h5 = manifest.get("pool_h5") + if isinstance(pool_h5, Mapping) and "materializer_version" in pool_h5: + markers.add("pool_h5.materializer_version") + return markers + + +def _validate_canonical_legacy_envelope( + manifest: Mapping[str, object], + *, + manifest_path: Path, +) -> None: + """Require positive identity for the frozen schema-4 publication route.""" + + failures: list[str] = [] + if manifest.get("operator_order") != list(_LEGACY_POOL_OPERATOR_ORDER): + failures.append("operator_order") + stage_receipts = manifest.get("stage_receipts") + if not isinstance(stage_receipts, Mapping) or not ( + _LEGACY_REQUIRED_STAGE_RECEIPTS <= set(stage_receipts) + ): + failures.append("stage_receipts") + checkpoints = manifest.get("stage_checkpoints") + if not isinstance(checkpoints, Mapping): + failures.append("stage_checkpoints") + else: + expected_checkpoint_identity = { + "artifact_kind": _LEGACY_POOL_CHECKPOINT_ARTIFACT_KIND, + "schema_version": _LEGACY_POOL_CHECKPOINT_SCHEMA_VERSION, + "materializer_version": _LEGACY_POOL_CHECKPOINT_MATERIALIZER_VERSION, + } + if any( + checkpoints.get(key) != value + for key, value in expected_checkpoint_identity.items() + ): + failures.append("stage_checkpoints.identity") + stages = checkpoints.get("stages") + if isinstance(stages, Mapping) and any( + not isinstance(receipt, Mapping) + or receipt.get("materializer_version") + != _LEGACY_POOL_CHECKPOINT_MATERIALIZER_VERSION + for receipt in stages.values() + ): + failures.append("stage_checkpoints.stages") + agreement = manifest.get("agreement_gate") + gates = agreement.get("gates") if isinstance(agreement, Mapping) else None + if not isinstance(gates, Mapping) or set(gates) != {"us_spine_agreement"}: + failures.append("agreement_gate.us_spine_agreement") + if failures: + raise ValueError( + f"US multispine pool manifest {manifest_path} is not a canonical " + f"legacy envelope; invalid={sorted(failures)}." + ) + + +def _validated_pool_manifest_envelope( + manifest: Mapping[str, object], + *, + manifest_path: Path, +) -> str: + """Classify only an unambiguous schema-bound legacy or stacked envelope.""" + + schema_version = manifest.get("schema_version") + markers = _stacked_manifest_markers(manifest) + if schema_version == US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION: + missing = _REQUIRED_STACKED_MANIFEST_FIELDS - set(manifest) + if manifest.get("pipeline") != _STACKED_PIPELINE or missing: + raise ValueError( + f"US multispine pool manifest {manifest_path} has an " + "ambiguous stacked envelope; " + f"missing={sorted(missing)}." + ) + return "stacked" + if schema_version == _LEGACY_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION: + if markers: + raise ValueError( + f"US multispine pool manifest {manifest_path} legacy envelope " + f"carries stacked-only field(s) {sorted(markers)}." + ) + _validate_canonical_legacy_envelope(manifest, manifest_path=manifest_path) + return "legacy" + raise ValueError( + f"US multispine pool manifest {manifest_path} has an unsupported " + "artifact binding." + ) class AuthenticatedPoolH5MismatchError(RuntimeError): @@ -146,17 +320,11 @@ def load_legacy_calibrated_us_h5(path: str | Path) -> Frame: multispine pool, whose importance-weight receipt lives in its manifest. """ - from policyengine_us.data import USSingleYearDataset - - dataset = USSingleYearDataset(file_path=str(Path(path))) - tables = { - "person": dataset.person, - "household": dataset.household.copy(), - "tax_unit": dataset.tax_unit, - "spm_unit": dataset.spm_unit, - "family": dataset.family, - "marital_unit": dataset.marital_unit, - } + with pd.HDFStore(Path(path), mode="r") as store: + tables = { + entity: read_frame_table(store, entity) for entity in US_SCHEMA.entities + } + tables["household"] = tables["household"].copy() household_weights = ( tables["household"].pop("household_weight").to_numpy(dtype=np.float64) ) @@ -212,10 +380,7 @@ def _load_authenticated_us_multispine_pool_manifest( label="pool manifest", expected_sha256=expected_manifest_sha256, ) - if ( - manifest.get("artifact_kind") != US_MULTISPINE_POOL_MANIFEST_ARTIFACT_KIND - or manifest.get("schema_version") != US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION - ): + if manifest.get("artifact_kind") != US_MULTISPINE_POOL_MANIFEST_ARTIFACT_KIND: raise ValueError( f"US multispine pool manifest {manifest_path} has an unsupported " "artifact binding." @@ -227,6 +392,19 @@ def _load_authenticated_us_multispine_pool_manifest( raise ValueError( f"US multispine pool manifest {manifest_path} is not simulation-ready." ) + envelope = _validated_pool_manifest_envelope( + manifest, + manifest_path=manifest_path, + ) + expected_schema_version = ( + US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION + if envelope == "stacked" + else _LEGACY_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION + ) + _validate_stacked_late_dag_manifest_binding( + manifest, + manifest_path=manifest_path, + ) checkpoint_provenance = _mapping( manifest.get("stage_checkpoints"), label=f"US multispine pool manifest {manifest_path}.stage_checkpoints", @@ -285,6 +463,29 @@ def _load_authenticated_us_multispine_pool_manifest( f"US multispine pool H5 {pool_path} publication run ID does not " "match its manifest." ) + if envelope == "stacked": + receipt_materializer = pool_receipt.get("materializer_version") + h5_materializer = h5_metadata.get("materializer_version") + if ( + type(receipt_materializer) is not int + or receipt_materializer != US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION + or type(h5_materializer) is not int + or h5_materializer != US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION + ): + raise ValueError( + f"US stacked pool publication {manifest_path} does not bind " + "the current H5 materializer version in both its manifest " + f"receipt and H5 metadata: receipt={receipt_materializer!r}, " + f"h5={h5_materializer!r}, expected=" + f"{US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION}." + ) + elif ( + "materializer_version" in pool_receipt or "materializer_version" in h5_metadata + ): + raise ValueError( + f"US multispine pool manifest {manifest_path} legacy envelope " + "carries a stacked-only H5 materializer version." + ) diagnostics_receipt = _mapping( manifest.get("agreement_diagnostics"), @@ -308,11 +509,31 @@ def _load_authenticated_us_multispine_pool_manifest( diagnostics_path, label="pool agreement diagnostics", ) + diagnostics_stacked_fields = set(diagnostics) & { + "pipeline", + "semantic_kind", + "release_id", + "terminal_gates", + } + if envelope == "stacked": + if ( + diagnostics.get("pipeline") != _STACKED_PIPELINE + or diagnostics.get("semantic_kind") != "stacked_terminal_gates" + ): + raise ValueError( + f"US multispine pool diagnostics {diagnostics_path} have an " + "ambiguous stacked envelope." + ) + elif diagnostics_stacked_fields: + raise ValueError( + f"US multispine pool diagnostics {diagnostics_path} legacy " + "envelope carries stacked-only field(s) " + f"{sorted(diagnostics_stacked_fields)}." + ) if ( diagnostics.get("artifact_kind") != US_MULTISPINE_AGREEMENT_DIAGNOSTICS_ARTIFACT_KIND - or diagnostics.get("schema_version") - != US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION + or diagnostics.get("schema_version") != expected_schema_version or diagnostics.get("simulation_ready") is not True or diagnostics.get("publication_run_id") != publication_run_id ): @@ -350,6 +571,104 @@ def _load_authenticated_us_multispine_pool_manifest( ) +def _validate_stacked_late_dag_manifest_binding( + manifest: Mapping[str, object], + *, + manifest_path: Path, +) -> None: + """Make schema-8 stacked consumers authenticate the published DAG proof.""" + + if manifest.get("pipeline") != "us-stacked-pool": + return + if manifest.get("operator_order") != list(US_STACKED_POOL_OPERATOR_ORDER): + raise ValueError( + f"US stacked pool manifest {manifest_path} does not bind the " + "canonical late-DAG operator order." + ) + stage_receipts = manifest.get("stage_receipts") + impute = ( + stage_receipts.get("impute") if isinstance(stage_receipts, Mapping) else None + ) + dag = ( + impute.get("stacked_late_producer_dag") if isinstance(impute, Mapping) else None + ) + if not isinstance(dag, Mapping): + raise ValueError( + f"US stacked pool manifest {manifest_path} has no late-producer " + "DAG receipt." + ) + from microcosm.build.us_runtime.stacked_spine import ( + validate_stacked_late_producer_receipt, + ) + + validate_stacked_late_producer_receipt( + dag, + boundary=f"US stacked pool manifest {manifest_path}", + ) + _stacked_late_transition_binding( + manifest, + manifest_path=manifest_path, + ) + transfer_alias = impute.get("stacked_post_puf_transfer") + source_chain = impute.get("source_operator_chain") + source_alias = ( + source_chain.get("late_dag_completion") + if isinstance(source_chain, Mapping) + else None + ) + if transfer_alias != dag.get("post_puf_transfer"): + raise ValueError( + f"US stacked pool manifest {manifest_path} post-PUF transfer alias " + "differs from its late-DAG proof." + ) + if source_alias != dag.get("source_completion"): + raise ValueError( + f"US stacked pool manifest {manifest_path} source-completion alias " + "differs from its late-DAG proof." + ) + + +def _stacked_late_transition_binding( + manifest: Mapping[str, object], + *, + manifest_path: Path, +) -> tuple[Mapping[str, object], Mapping[str, object], str] | None: + """Return the signed DAG, derived authority, and independent authority SHA.""" + + if manifest.get("pipeline") != "us-stacked-pool": + return None + stage_receipts = manifest.get("stage_receipts") + impute = ( + stage_receipts.get("impute") if isinstance(stage_receipts, Mapping) else None + ) + dag = ( + impute.get("stacked_late_producer_dag") if isinstance(impute, Mapping) else None + ) + if not isinstance(dag, Mapping): + raise ValueError( + f"US stacked pool manifest {manifest_path} has no late-producer " + "DAG receipt." + ) + from microcosm.build.us_runtime.stacked_spine import ( + _late_producer_transition_authority_receipt, + ) + + derived_authority = _late_producer_transition_authority_receipt(dag) + expected_sha256 = derived_authority["sha256"] + observed_sha256 = manifest.get("late_producer_transition_authority_sha256") + if ( + not isinstance(observed_sha256, str) + or _LOWERCASE_SHA256.fullmatch(observed_sha256) is None + or observed_sha256 != expected_sha256 + ): + raise ValueError( + f"US stacked pool manifest {manifest_path} independently carried " + "late-producer transition authority does not match its signed DAG " + f"receipt; expected={expected_sha256!r}, observed={observed_sha256!r}." + ) + return dag, derived_authority, observed_sha256 + + def load_simulation_ready_us_multispine_pool( path: str | Path, *, @@ -405,7 +724,7 @@ def load_simulation_ready_us_multispine_pool( ) tables = { entity: canonicalize_table_string_dtypes( - store[entity], + read_frame_table(store, entity), boundary="simulation-ready US pool H5 load", table_name=entity, ) @@ -426,6 +745,18 @@ def load_simulation_ready_us_multispine_pool( ) household_weights = household.pop("household_weight").to_numpy(dtype=np.float64) tables["household"] = household + late_transition = _stacked_late_transition_binding( + manifest, + manifest_path=manifest_path, + ) + frame_metadata: dict[str, object] = {} + if late_transition is not None: + _dag, transition_authority, _transition_authority_sha256 = late_transition + from microcosm.build.us_runtime.stacked_spine import ( + US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY, + ) + + frame_metadata[US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY] = transition_authority frame = Frame( tables, US_SCHEMA, @@ -435,7 +766,20 @@ def load_simulation_ready_us_multispine_pool( WeightKind.IMPORTANCE, ) }, + metadata=frame_metadata, ) + if late_transition is not None: + dag, _transition_authority, transition_authority_sha256 = late_transition + from microcosm.build.us_runtime.stacked_spine import ( + validate_stacked_late_producer_transition_authority, + ) + + validate_stacked_late_producer_transition_authority( + frame, + dag, + boundary=f"US stacked pool H5 {pool_path}", + expected_transition_authority_sha256=transition_authority_sha256, + ) provenance_counts = _mapping( manifest.get("provenance_counts"), @@ -504,6 +848,7 @@ def write_nullable_us_h5( period: int, artifact_kind: str, publication_run_id: str | None = None, + materializer_version: int | None = None, ) -> None: """Atomically write and verify a nullable US single-year H5. @@ -521,6 +866,10 @@ def write_nullable_us_h5( not isinstance(publication_run_id, str) or not publication_run_id.strip() ): raise ValueError("publication_run_id must be a non-empty string when set.") + if materializer_version is not None and ( + type(materializer_version) is not int or materializer_version <= 0 + ): + raise ValueError("materializer_version must be a positive integer when set.") for entity in US_SCHEMA.entities: canonicalize_table_string_dtypes( @@ -540,6 +889,7 @@ def write_nullable_us_h5( period=int(period), artifact_kind=artifact_kind, publication_run_id=publication_run_id, + materializer_version=materializer_version, ) _verify_nullable_us_h5( frame, @@ -547,6 +897,7 @@ def write_nullable_us_h5( period=int(period), artifact_kind=artifact_kind, publication_run_id=publication_run_id, + materializer_version=materializer_version, ) os.replace(temporary, output) except BaseException: @@ -561,18 +912,19 @@ def _write_nullable_us_h5_file( period: int, artifact_kind: str, publication_run_id: str | None, + materializer_version: int | None, ) -> None: with pd.HDFStore(path, mode="w") as store: for entity in frame.entities: table = _export_table(frame, entity) if not len(table): continue - # Fixed format preserves mixed bool/null object columns - # losslessly. Table format rejects them, which would force a - # fill or type rewrite. - with warnings.catch_warnings(): - warnings.simplefilter("ignore", pd.errors.PerformanceWarning) - store.put(entity, table, format="fixed") + put_frame_table( + store, + entity, + table, + preferred_format="fixed", + ) store.put( _TIME_PERIOD_KEY, pd.Series([period]), @@ -587,6 +939,7 @@ def _write_nullable_us_h5_file( frame, artifact_kind=artifact_kind, publication_run_id=publication_run_id, + materializer_version=materializer_version, ), sort_keys=True, ) @@ -603,15 +956,17 @@ def _verify_nullable_us_h5( period: int, artifact_kind: str, publication_run_id: str | None, + materializer_version: int | None, ) -> None: with pd.HDFStore(path, mode="r") as store: for entity in frame.entities: expected = _export_table(frame, entity) if not len(expected): continue + expected = materialize_nullable_booleans_for_pytables(expected).table try: stored = canonicalize_table_string_dtypes( - store[entity], + read_frame_table(store, entity), boundary="nullable US H5 verification load", table_name=entity, ) @@ -655,6 +1010,7 @@ def _verify_nullable_us_h5( frame, artifact_kind=artifact_kind, publication_run_id=publication_run_id, + materializer_version=materializer_version, ) if stored_metadata != expected_metadata: raise RuntimeError( @@ -681,7 +1037,8 @@ def _artifact_metadata( *, artifact_kind: str, publication_run_id: str | None, -) -> dict[str, str]: + materializer_version: int | None, +) -> dict[str, object]: metadata = { "artifact_kind": artifact_kind, "entity_hdf_format": "fixed_nullable", @@ -689,6 +1046,8 @@ def _artifact_metadata( } if publication_run_id is not None: metadata["publication_run_id"] = publication_run_id + if materializer_version is not None: + metadata["materializer_version"] = materializer_version return metadata diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/housing_inputs.py b/packages/microcosm-build/src/microcosm/build/us_runtime/housing_inputs.py index 83522186..1f83bd82 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/housing_inputs.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/housing_inputs.py @@ -53,6 +53,8 @@ "US_HOUSING_HOUSEHOLD_OUTPUT_COLUMNS", "US_HOUSING_INPUTS_OUTPUT_COLUMNS", "US_HOUSING_INPUTS_STAGE_NAME", + "US_HOUSING_ASSISTANCE_PUF_MAX_TRAIN_SAMPLES", + "US_HOUSING_ASSISTANCE_PUF_N_ESTIMATORS", "US_HOUSING_NONCONSTANT_HOUSEHOLD_COLUMNS", "US_HOUSING_NONCONSTANT_PERSON_COLUMNS", "US_HOUSING_NONCONSTANT_SPM_UNIT_COLUMNS", @@ -166,7 +168,8 @@ _DONOR_REAL_ESTATE_TAX_ALLOCATION_COLUMN = "real_estate_taxes_is_allocated" _MAX_TRAIN_SAMPLES = 10_000 _DEFAULT_N_ESTIMATORS = 100 -_PUF_MAX_TRAIN_SAMPLES = 5_000 +US_HOUSING_ASSISTANCE_PUF_N_ESTIMATORS = 100 +US_HOUSING_ASSISTANCE_PUF_MAX_TRAIN_SAMPLES = 5_000 _RENT_SHARE_BAND = (0.05, 0.25) _HOUSING_ASSISTANCE_SHARE_BAND = (0.005, 0.08) @@ -982,8 +985,8 @@ def impute_us_housing_assistance_to_puf_support( frame: Frame, *, seed: int, - n_estimators: int = _DEFAULT_N_ESTIMATORS, - max_train_samples: int = _PUF_MAX_TRAIN_SAMPLES, + n_estimators: int = US_HOUSING_ASSISTANCE_PUF_N_ESTIMATORS, + max_train_samples: int = US_HOUSING_ASSISTANCE_PUF_MAX_TRAIN_SAMPLES, ) -> Frame: """Replace only the PUF clone's housing-assistance receipt flag by QRF. diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/late_producer_dag.py b/packages/microcosm-build/src/microcosm/build/us_runtime/late_producer_dag.py new file mode 100644 index 00000000..a3554aa7 --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/late_producer_dag.py @@ -0,0 +1,518 @@ +"""Deterministic producer-input DAG primitives for the US late stage. + +The graph is deliberately data-only. Country-specific code declares which +stage produces each scoped input; this module validates those declarations, +derives a stable topological schedule, names cycles, and fences callback +execution when a required input remains unfilled. +""" + +from __future__ import annotations + +import hashlib +import json +from collections.abc import Callable, Mapping +from dataclasses import dataclass + +__all__ = [ + "ProducerInputColumn", + "ProducerContract", + "ProducerInput", + "ProducerOutput", + "ProducerSchedule", + "derive_producer_schedule", + "run_producer_when_ready", +] + + +# A whole-pool producer satisfies either canonical row subset. Subset +# producers cannot satisfy one another or a whole-pool consumer. ``receipt`` +# is the non-row output emitted by each source operator; availability of that +# stage receipt satisfies the source finalizer's whole-pool readiness gate. +# Unlisted extension scopes are deliberately exact-match only. +_PRODUCER_SCOPE_COVERAGE: Mapping[str, frozenset[str]] = { + "whole_pool": frozenset({"whole_pool", "asec_source", "acs_source", "puf_clone"}), + "asec_source": frozenset({"asec_source"}), + "acs_source": frozenset({"acs_source"}), + "puf_clone": frozenset({"puf_clone"}), + "receipt": frozenset({"whole_pool"}), +} + + +def _nonempty(value: object, *, label: str) -> str: + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"{label} must be a non-empty string.") + return value + + +@dataclass(frozen=True, order=True) +class ProducerInputColumn: + """One physical column participating in an effective input alternative.""" + + entity: str + column: str + value_kind: str = "non_null" + + def __post_init__(self) -> None: + _nonempty(self.entity, label="ProducerInputColumn.entity") + _nonempty(self.column, label="ProducerInputColumn.column") + if self.value_kind not in { + "column_present", + "non_null", + "finite_numeric", + }: + raise ValueError( + "ProducerInputColumn.value_kind must be 'column_present', " + f"'non_null', or 'finite_numeric'; got {self.value_kind!r}." + ) + + +@dataclass(frozen=True, order=True) +class ProducerInput: + """One scoped input and the stage expected to make it ready.""" + + entity: str + column: str + required_scope: str + producing_stage: str + tolerated_absence_receipts: tuple[str, ...] = () + alternatives: tuple[tuple[ProducerInputColumn, ...], ...] = () + + def __post_init__(self) -> None: + for label, value in ( + ("ProducerInput.entity", self.entity), + ("ProducerInput.column", self.column), + ("ProducerInput.required_scope", self.required_scope), + ("ProducerInput.producing_stage", self.producing_stage), + ): + _nonempty(value, label=label) + receipts = tuple(self.tolerated_absence_receipts) + if any(not isinstance(item, str) or not item.strip() for item in receipts): + raise ValueError( + "ProducerInput.tolerated_absence_receipts must contain only " + "non-empty strings." + ) + if len(set(receipts)) != len(receipts): + raise ValueError( + "ProducerInput.tolerated_absence_receipts contains duplicates." + ) + object.__setattr__(self, "tolerated_absence_receipts", tuple(sorted(receipts))) + alternatives = tuple(tuple(option) for option in self.alternatives) + if not alternatives: + alternatives = ((ProducerInputColumn(self.entity, self.column),),) + if any(not option for option in alternatives) or any( + not isinstance(item, ProducerInputColumn) + for option in alternatives + for item in option + ): + raise TypeError( + "ProducerInput.alternatives require nonempty tuples of " + "ProducerInputColumn values." + ) + canonical_alternatives = tuple( + sorted( + {tuple(sorted(set(option))) for option in alternatives}, + key=lambda option: tuple( + (item.entity, item.column, item.value_kind) for item in option + ), + ) + ) + object.__setattr__(self, "alternatives", canonical_alternatives) + + +@dataclass(frozen=True, order=True) +class ProducerOutput: + """One column and the row scope covered by its producing stage.""" + + entity: str + column: str + coverage_scope: str + + def __post_init__(self) -> None: + for label, value in ( + ("ProducerOutput.entity", self.entity), + ("ProducerOutput.column", self.column), + ("ProducerOutput.coverage_scope", self.coverage_scope), + ): + _nonempty(value, label=label) + + +@dataclass(frozen=True) +class ProducerContract: + """One executable producer and its complete declared graph surface.""" + + name: str + kind: str + inputs: tuple[ProducerInput, ...] + outputs: tuple[ProducerOutput, ...] + + def __post_init__(self) -> None: + _nonempty(self.name, label="ProducerContract.name") + _nonempty(self.kind, label="ProducerContract.kind") + inputs = tuple(self.inputs) + outputs = tuple(self.outputs) + if any(not isinstance(item, ProducerInput) for item in inputs): + raise TypeError("ProducerContract.inputs require ProducerInput values.") + if any(not isinstance(item, ProducerOutput) for item in outputs): + raise TypeError("ProducerContract.outputs require ProducerOutput values.") + if len(set(inputs)) != len(inputs): + raise ValueError(f"Late producer {self.name!r} repeats an input.") + if len(set(outputs)) != len(outputs): + raise ValueError(f"Late producer {self.name!r} repeats an output.") + object.__setattr__(self, "inputs", tuple(sorted(inputs))) + object.__setattr__(self, "outputs", tuple(sorted(outputs))) + + +@dataclass(frozen=True) +class ProducerSchedule: + """Canonical topological waves and their byte-stable identity.""" + + order: tuple[str, ...] + waves: tuple[tuple[str, ...], ...] + edges: tuple[tuple[str, str], ...] + canonical_json: bytes + sha256: str + + +def _contract_payload(contract: ProducerContract) -> dict[str, object]: + return { + "name": contract.name, + "kind": contract.kind, + "inputs": [ + { + "entity": item.entity, + "column": item.column, + "required_scope": item.required_scope, + "producing_stage": item.producing_stage, + "tolerated_absence_receipts": list(item.tolerated_absence_receipts), + "alternatives": [ + [ + { + "entity": column.entity, + "column": column.column, + "value_kind": column.value_kind, + } + for column in alternative + ] + for alternative in item.alternatives + ], + } + for item in contract.inputs + ], + "outputs": [ + { + "entity": item.entity, + "column": item.column, + "coverage_scope": item.coverage_scope, + } + for item in contract.outputs + ], + } + + +def _named_cycle( + adjacency: Mapping[str, set[str]], + remaining: set[str], +) -> tuple[str, ...]: + """Return the first deterministic DFS cycle from a Kahn residual.""" + + state: dict[str, int] = {} + stack: list[str] = [] + stack_positions: dict[str, int] = {} + + def visit(node: str) -> tuple[str, ...] | None: + state[node] = 1 + stack_positions[node] = len(stack) + stack.append(node) + for child in sorted(adjacency[node] & remaining): + if state.get(child, 0) == 0: + found = visit(child) + if found is not None: + return found + elif state.get(child) == 1: + start = stack_positions[child] + return (*stack[start:], child) + stack.pop() + stack_positions.pop(node) + state[node] = 2 + return None + + for node in sorted(remaining): + if state.get(node, 0) == 0: + found = visit(node) + if found is not None: + return found + raise AssertionError("A nonempty Kahn residual did not contain a cycle.") + + +def _producer_scope_covers(output_scope: str, required_scope: str) -> bool: + """Return whether one declared output scope satisfies a consumer scope.""" + + covered = _PRODUCER_SCOPE_COVERAGE.get(output_scope) + if covered is None: + return output_scope == required_scope + return required_scope in covered + + +def derive_producer_schedule( + registry: Mapping[str, ProducerContract], + *, + external_stages: tuple[str, ...] = (), +) -> ProducerSchedule: + """Validate declarations and derive deterministic topological waves.""" + + if not isinstance(registry, Mapping): + raise TypeError("Late producer registry must be a mapping.") + external = tuple(external_stages) + if any(not isinstance(stage, str) or not stage.strip() for stage in external): + raise ValueError("External producer stages must be non-empty strings.") + if len(set(external)) != len(external): + raise ValueError("External producer stages contain duplicates.") + contracts = dict(registry) + invalid_values = sorted( + name + for name, contract in contracts.items() + if not isinstance(contract, ProducerContract) + ) + if invalid_values: + raise TypeError( + "Late producer registry values must be ProducerContract instances: " + f"{invalid_values}." + ) + mismatched = sorted( + (name, contract.name) + for name, contract in contracts.items() + if name != contract.name + ) + if mismatched: + raise ValueError( + f"Late producer registry keys must equal contract names: {mismatched}." + ) + overlap = sorted(set(contracts) & set(external)) + if overlap: + raise ValueError( + f"Late producer stages cannot also be external stages: {overlap}." + ) + + adjacency = {name: set() for name in contracts} + indegree = {name: 0 for name in contracts} + edges: set[tuple[str, str]] = set() + unknown: list[tuple[str, str]] = [] + missing_outputs: list[tuple[str, str, str]] = [] + scope_mismatches: list[tuple[str, str, str, str, tuple[str, ...]]] = [] + for consumer_name in sorted(contracts): + contract = contracts[consumer_name] + for item in contract.inputs: + producer_name = item.producing_stage + if producer_name in external: + continue + producer = contracts.get(producer_name) + if producer is None: + unknown.append((consumer_name, producer_name)) + continue + matching_outputs = tuple( + output + for output in producer.outputs + if output.entity == item.entity and output.column == item.column + ) + if not matching_outputs: + missing_outputs.append( + (consumer_name, producer_name, f"{item.entity}.{item.column}") + ) + continue + if not any( + _producer_scope_covers( + output.coverage_scope, + item.required_scope, + ) + for output in matching_outputs + ): + scope_mismatches.append( + ( + consumer_name, + producer_name, + f"{item.entity}.{item.column}", + item.required_scope, + tuple( + sorted( + {output.coverage_scope for output in matching_outputs} + ) + ), + ) + ) + continue + edge = (producer_name, consumer_name) + if edge not in edges: + edges.add(edge) + adjacency[producer_name].add(consumer_name) + indegree[consumer_name] += 1 + if unknown or missing_outputs or scope_mismatches: + raise ValueError( + "Late producer dependency declarations are invalid; " + f"unknown_stages={unknown}, missing_outputs={missing_outputs}, " + f"scope_mismatches={scope_mismatches}." + ) + + remaining = set(contracts) + waves: list[tuple[str, ...]] = [] + while remaining: + ready = tuple(sorted(name for name in remaining if indegree[name] == 0)) + if not ready: + cycle = _named_cycle(adjacency, remaining) + raise RuntimeError( + "Late producer dependency cycle: " + " -> ".join(cycle) + "." + ) + waves.append(ready) + for producer_name in ready: + remaining.remove(producer_name) + for producer_name in ready: + for consumer_name in adjacency[producer_name]: + indegree[consumer_name] -= 1 + + order = tuple(name for wave in waves for name in wave) + sorted_edges = tuple(sorted(edges)) + payload = { + "schema_version": 2, + "external_stages": sorted(external), + "scope_coverage": { + "declared": { + output_scope: sorted(required_scopes) + for output_scope, required_scopes in sorted( + _PRODUCER_SCOPE_COVERAGE.items() + ) + }, + "unlisted_scope_rule": "exact_match_only", + }, + "contracts": [_contract_payload(contracts[name]) for name in sorted(contracts)], + "edges": [list(edge) for edge in sorted_edges], + "waves": [list(wave) for wave in waves], + "order": list(order), + } + canonical_json = json.dumps( + payload, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + return ProducerSchedule( + order=order, + waves=tuple(waves), + edges=sorted_edges, + canonical_json=canonical_json, + sha256=hashlib.sha256(canonical_json).hexdigest(), + ) + + +def _absence_receipt_matches( + receipt_id: str, + receipt: object, + requirement: ProducerInput, + rows: int, + *, + producer: str, +) -> bool: + return bool( + isinstance(receipt, Mapping) + and receipt.get("receipt_id") == receipt_id + and receipt.get("status") == "declared_absence" + and receipt.get("entity") == requirement.entity + and receipt.get("column") == requirement.column + and receipt.get("required_scope") == requirement.required_scope + and receipt.get("rows") == rows + and receipt.get("producer") == producer + and receipt.get("reason") == "optional availability-pattern input" + ) + + +def run_producer_when_ready[ResultT]( + contract: ProducerContract, + callback: Callable[[], ResultT], + *, + unfilled_rows: Mapping[ProducerInput, int], + invalid_rows: Mapping[ProducerInput, int], + absence_receipts: Mapping[str, Mapping[str, object]], +) -> ResultT: + """Fence one producer callback on exact input or absence evidence. + + Missing values and invalid values are separate states. Only missing + values can be authorized by a declared-absence receipt; nonnumeric or + nonfinite values always fail closed. + """ + + if not isinstance(contract, ProducerContract): + raise TypeError("Producer readiness requires a ProducerContract.") + if not callable(callback): + raise TypeError(f"Late producer {contract.name!r} callback is not callable.") + declared_inputs = set(contract.inputs) + + def sort_key(item: ProducerInput) -> tuple[str, str, str, str]: + return ( + item.entity, + item.column, + item.required_scope, + item.producing_stage, + ) + + missing_unfilled = sorted(declared_inputs - set(unfilled_rows), key=sort_key) + unexpected_unfilled = sorted(set(unfilled_rows) - declared_inputs, key=sort_key) + if missing_unfilled or unexpected_unfilled: + raise ValueError( + f"Late producer {contract.name!r} unfilled-row readiness surface " + f"drifted; missing={missing_unfilled}, extra={unexpected_unfilled}." + ) + missing_invalid = sorted(declared_inputs - set(invalid_rows), key=sort_key) + unexpected_invalid = sorted(set(invalid_rows) - declared_inputs, key=sort_key) + if missing_invalid or unexpected_invalid: + raise ValueError( + f"Late producer {contract.name!r} invalid-value readiness surface " + f"drifted; missing={missing_invalid}, extra={unexpected_invalid}." + ) + failures: list[str] = [] + for requirement in contract.inputs: + rows = unfilled_rows[requirement] + if isinstance(rows, bool) or not isinstance(rows, int) or rows < 0: + raise ValueError( + f"Late producer {contract.name!r} unfilled count for " + f"{requirement.entity}.{requirement.column} must be a " + f"non-negative integer; got {rows!r}." + ) + invalid = invalid_rows[requirement] + if isinstance(invalid, bool) or not isinstance(invalid, int) or invalid < 0: + raise ValueError( + f"Late producer {contract.name!r} invalid count for " + f"{requirement.entity}.{requirement.column} must be a " + f"non-negative integer; got {invalid!r}." + ) + if invalid: + failures.append( + f"{requirement.entity}.{requirement.column}: {invalid} invalid " + f"value(s) in required scope {requirement.required_scope!r}; " + f"declared producing stage is {requirement.producing_stage!r}; " + "declared absence cannot authorize invalid values." + ) + if rows == 0: + continue + tolerated = any( + _absence_receipt_matches( + receipt_id, + absence_receipts.get(receipt_id), + requirement, + rows, + producer=contract.name, + ) + for receipt_id in requirement.tolerated_absence_receipts + ) + if tolerated: + continue + allowed = list(requirement.tolerated_absence_receipts) + failures.append( + f"{requirement.entity}.{requirement.column}: {rows} unfilled row(s) " + f"in required scope {requirement.required_scope!r}; declared " + f"producing stage is {requirement.producing_stage!r}; tolerated " + f"absence receipts={allowed}." + ) + if failures: + raise ValueError( + f"Late producer {contract.name!r} refused unfilled input(s):\n " + + "\n ".join(failures) + ) + return callback() diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/multispine_pool.py b/packages/microcosm-build/src/microcosm/build/us_runtime/multispine_pool.py index 4ffbdb0b..52401171 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/multispine_pool.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/multispine_pool.py @@ -13,8 +13,12 @@ from __future__ import annotations +import hashlib +import json from collections.abc import Callable, Mapping from dataclasses import dataclass, field +from functools import lru_cache +from importlib.metadata import version from typing import Protocol import numpy as np @@ -22,6 +26,9 @@ from microcosm.build.gates import GateResult from microcosm.build.serialization_dtypes import canonicalize_frame_string_dtypes +from microcosm.build.us_runtime.acs_income_universe import ( + ACS_PUMS_EARNINGS_UNIVERSE_PERSON_INPUTS, +) from microcosm.build.us_runtime.acs_transfer import ( ACS_NATIVE_PERSON_INPUTS, TargetFamilies, @@ -48,6 +55,8 @@ with_us_hours_worked_inputs, ) from microcosm.build.us_runtime.housing_inputs import ( + US_HOUSING_ASSISTANCE_PUF_MAX_TRAIN_SAMPLES, + US_HOUSING_ASSISTANCE_PUF_N_ESTIMATORS, impute_us_housing_assistance_to_puf_support, with_us_housing_inputs, ) @@ -66,6 +75,7 @@ from microcosm.build.us_runtime.puf_qrf_chain import PRIMARY_QRF_TARGET_ORDER from microcosm.build.us_runtime.puf_support import clone_us_frame_for_puf_support from microcosm.build.us_runtime.qbi_inputs import ( + US_QBI_RECONCILED_PERSON_COLUMNS, bind_us_qbi_reconciliation_transition_authority, us_qbi_post_reconciliation_person_columns, us_qbi_reconciliation_change_receipt, @@ -92,6 +102,7 @@ spine_assembly_receipt, spine_provenance_counts, support_clone_index_column, + support_source_id_column, validate_assembly_provenance, without_support_role_metadata, ) @@ -100,38 +111,63 @@ TakeUpProgram, load_take_up_contract, ) +from microcosm.build.us_runtime.us_late_overlap_ownership import ( + US_LATE_EDUCATION_NOOP_TARGETS, + US_LATE_RETIREMENT_SOURCE_MIRROR_TARGETS, + us_late_overlap_ownership_receipt, +) from microcosm.build.us_runtime.weeks_unemployed import with_us_weeks_unemployed from microcosm.build.us_runtime.wic_claim import with_us_wic_claim_input from microcosm.build.us_runtime.workers_compensation import ( with_us_workers_compensation, ) -from microcosm.frame import Frame +from microcosm.frame import US_SCHEMA, Frame +from microcosm.frame.adapters.policyengine_us import ( + PolicyEngineUSVariableMetadataIndex, + VariableDependencyClosure, +) __all__ = [ "POOL_CHECKPOINT_STAGE_ORDER", + "POOL_ENGINE_INPUT_PROJECTION_CONTRACT", "POOL_HOUSEHOLD_MASS_SHARES", + "POOL_HOUSING_ASSISTANCE_MAX_TRAIN_SAMPLES", + "POOL_HOUSING_ASSISTANCE_N_ESTIMATORS", "POOL_DERIVE_OPERATOR_ORDER", "POOL_DEFERRED_TRANSFER_INPUTS", + "POOL_DEFERRED_TRANSFER_STATUS", "POOL_OPERATOR_CONTRACTS", "POOL_OPERATOR_ORDER", "POOL_RANDOM_SEED", + "POOL_REMAINING_STAGE_INPUT_MANIFEST_SHA256", + "POOL_SSI_DEPENDENCY_CONTRACT", "POOL_SIMULATION_HOUSEHOLD_BATCH_SIZE", "POOL_POST_CLONE_SOURCE_OPERATOR_ORDER", + "POOL_POST_CLONE_SOURCE_PHASE", "POOL_PRE_CLONE_SOURCE_OPERATOR_ORDER", "POOL_SOURCE_OPERATOR_CONTRACTS", "POOL_SOURCE_OPERATOR_ORDER", + "POOL_SOURCE_ALLOW_EXISTING_WITHOUT_SOURCE", "POOL_SPINE_AGREEMENT_REGISTRY", "POOL_TIME_PERIOD", "MultispinePoolCheckpoint", "MultispinePoolResult", "PoolInputSurfaceEntry", + "PoolEngineInputProjectionContract", + "PoolRemainingStageInput", + "PoolSsiDependencyContract", "PoolStageOutput", "SourceOperatorContract", "complete_multispine_source_inputs", "derive_multispine_pool_inputs", + "finalize_multispine_source_inputs", "materialize_multispine_agreement_outputs", "materialize_pool_deferred_transfer_inputs", "pool_input_surface", + "pool_engine_input_projection_receipt", + "pool_remaining_stage_input_manifest", + "pool_remaining_stage_input_manifest_receipt", + "pool_ssi_dependency_closure", "pool_post_puf_puf_producer_target_families", "pool_post_puf_source_producer_target_families", "pool_post_puf_transfer_target_families", @@ -139,6 +175,7 @@ "pool_transfer_target_families", "prepare_multispine_puf_predictors", "prepare_multispine_source_inputs_for_clone", + "run_multispine_post_clone_source_operator", "run_multispine_pool_path", "seed_multispine_pool_inputs", ] @@ -204,6 +241,13 @@ POOL_TIME_PERIOD = 2024 """PolicyEngine period of the 2024 source pool.""" +POOL_SOURCE_ALLOW_EXISTING_WITHOUT_SOURCE = False +"""Source construction never reuses an unreceipted pre-existing surface.""" + +POOL_HOUSING_ASSISTANCE_N_ESTIMATORS = US_HOUSING_ASSISTANCE_PUF_N_ESTIMATORS +POOL_HOUSING_ASSISTANCE_MAX_TRAIN_SAMPLES = US_HOUSING_ASSISTANCE_PUF_MAX_TRAIN_SAMPLES +"""Exact direct-QRF controls for the housing-assistance source producer.""" + POOL_SIMULATION_HOUSEHOLD_BATCH_SIZE = 5_000 """Fixed household batch size for terminal formula-output evaluation.""" @@ -239,6 +283,73 @@ class PoolInputSurfaceEntry: provenance: tuple[str, ...] +@dataclass(frozen=True, order=True) +class PoolRemainingStageInput: + """One statically declared read after the transferred checkpoint. + + ``provision`` names the producer or fallback doctrine that makes the read + valid by ``available_by``. Pseudo-columns enclosed in angle brackets are + structural Frame resources rather than persisted PolicyEngine variables. + """ + + stage: str + consumer: str + entity: str + variable: str + execution_scope: str + provision: str + available_by: str + fallback: str | None = None + + +@dataclass(frozen=True) +class PoolSsiDependencyContract: + """Checked-in identity of the static PE-US SSI dependency closure.""" + + engine_version: str + root: str + input_leaf_count: int + formula_node_count: int + edge_count: int + sha256: str + + +@dataclass(frozen=True) +class PoolEngineInputProjectionContract: + """Pinned identity of every installed engine input scanned at simulate.""" + + engine_version: str + input_count: int + default_count: int + sha256: str + defaults_sha256: str + + +POOL_SSI_DEPENDENCY_CONTRACT = PoolSsiDependencyContract( + engine_version="1.764.6", + root="ssi", + input_leaf_count=55, + formula_node_count=62, + edge_count=186, + sha256="e3351cdedbe592456b637286ecd04b7079746e1c409e594fbca60a7d28666838", +) +"""Exact static graph consumed by the terminal SSI agreement simulation.""" + +POOL_ENGINE_INPUT_PROJECTION_CONTRACT = PoolEngineInputProjectionContract( + engine_version="1.764.6", + input_count=863, + default_count=863, + sha256="67a66b018c6261a03a88852cce5c5a4cbe9f5595735d17f2f7666e19e464dfbf", + defaults_sha256="87f508fbb382036946aa5e225d339e1b593a464ee6cfc644d7d710540b00a9a7", +) +"""Exact installed input registry scanned by the disposable projection.""" + +POOL_REMAINING_STAGE_INPUT_MANIFEST_SHA256 = ( + "8247a93e5f8f63d3ae71c1de681c29524d4bb8f07e3c6a50dcaf431b1377020f" +) +"""Pinned content digest of all 993 post-transfer consumer/input rows.""" + + @dataclass(frozen=True) class PoolStageOutput: """One source-blind operator result and its manifest-ready receipt.""" @@ -280,6 +391,7 @@ class MultispinePoolCheckpoint: stage_receipts: Mapping[str, Mapping[str, object]] simulation_frame: Frame | None = None qbi_transition_authority_sha256: str | None = None + late_producer_transition_authority_sha256: str | None = None def __post_init__(self) -> None: if self.stage not in POOL_CHECKPOINT_STAGE_ORDER: @@ -323,6 +435,18 @@ def __post_init__(self) -> None: "MultispinePoolCheckpoint.qbi_transition_authority_sha256 must " "be a string when present." ) + if ( + self.late_producer_transition_authority_sha256 is not None + and not isinstance( + self.late_producer_transition_authority_sha256, + str, + ) + ): + raise TypeError( + "MultispinePoolCheckpoint." + "late_producer_transition_authority_sha256 must be a string " + "when present." + ) @dataclass(frozen=True) @@ -360,6 +484,7 @@ class SourceOperatorContract: _PRE_CLONE_PHASE = "pre_clone" _POST_CLONE_PHASE = "post_clone" +POOL_POST_CLONE_SOURCE_PHASE = _POST_CLONE_PHASE _CPS_SOURCE_EXECUTION_SCOPE = "cps_source" _WHOLE_POOL_EXECUTION_SCOPE = "whole_pool" @@ -518,6 +643,20 @@ class SourceOperatorContract: "household": frozenset({"tenure_type"}), "spm_unit": frozenset({"spm_unit_tenure_type"}), } +_POOL_SIMULATION_PRESERVED_ENGINE_INPUTS: Mapping[ + tuple[str, str], tuple[str, str | None] +] = { + ("person", "is_related_to_head_or_spouse"): ("assembled", None), + ("household", "puma"): ( + "assembled", + "ephemeral_simulation_projection_engine_default_for_null", + ), + ("person", "ssi_reported"): ( + "transferred", + "ephemeral_simulation_projection_engine_default_for_null", + ), + ("household", "state_fips"): ("assembled", None), +} _SCF_WEALTH_DEFERRAL_REASON = ( "The increment-2 pool input contract contains no SCF 2022 or SIPP 2023 " @@ -539,6 +678,7 @@ class SourceOperatorContract: "stock_assets", ) } +POOL_DEFERRED_TRANSFER_STATUS = "deferred_pending_source_donor" """Pool-stage-only deferrals for source inputs whose donors are out of scope. These remain hard release requirements and remain in the legacy ACS transfer @@ -802,6 +942,689 @@ def register( ) +def pool_ssi_dependency_closure( + metadata_index: PolicyEngineUSVariableMetadataIndex | None = None, +) -> VariableDependencyClosure: + """Return SSI's static PE-US graph after checking the pinned identity.""" + + index = ( + metadata_index + if metadata_index is not None + else PolicyEngineUSVariableMetadataIndex() + ) + closure = index.variable_dependency_closure(POOL_SSI_DEPENDENCY_CONTRACT.root) + observed = { + "engine_version": closure.engine_version, + "root": closure.root, + "input_leaf_count": len(closure.input_leaves), + "formula_node_count": len(closure.formula_nodes), + "edge_count": len(closure.edges), + "sha256": closure.sha256, + } + expected = { + "engine_version": POOL_SSI_DEPENDENCY_CONTRACT.engine_version, + "root": POOL_SSI_DEPENDENCY_CONTRACT.root, + "input_leaf_count": POOL_SSI_DEPENDENCY_CONTRACT.input_leaf_count, + "formula_node_count": POOL_SSI_DEPENDENCY_CONTRACT.formula_node_count, + "edge_count": POOL_SSI_DEPENDENCY_CONTRACT.edge_count, + "sha256": POOL_SSI_DEPENDENCY_CONTRACT.sha256, + } + if observed != expected: + raise ValueError( + "PolicyEngine-US SSI dependency closure drifted; refresh the " + f"remaining-stage input audit. expected={expected}, observed={observed}." + ) + return closure + + +def _pool_engine_input_projection( + metadata_index: PolicyEngineUSVariableMetadataIndex, + *, + engine_version: str, +) -> tuple[tuple[str, str], ...]: + """Return every installed engine input after checking its pinned digest.""" + + projection = tuple( + (metadata_index.variable_metadata(variable).entity, variable) + for variable in metadata_index.variables() + ) + payload = [ + {"entity": entity, "variable": variable} for entity, variable in projection + ] + digest = hashlib.sha256( + json.dumps( + payload, + allow_nan=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + ).hexdigest() + observed = { + "engine_version": engine_version, + "input_count": len(projection), + "sha256": digest, + } + expected = { + "engine_version": POOL_ENGINE_INPUT_PROJECTION_CONTRACT.engine_version, + "input_count": POOL_ENGINE_INPUT_PROJECTION_CONTRACT.input_count, + "sha256": POOL_ENGINE_INPUT_PROJECTION_CONTRACT.sha256, + } + if observed != expected: + raise ValueError( + "PolicyEngine-US simulation input projection drifted; refresh the " + f"remaining-stage input audit. expected={expected}, observed={observed}." + ) + return projection + + +def pool_engine_input_projection_receipt( + engine: _PoolRulesEngine | None = None, +) -> dict[str, object]: + """Validate every installed simulation input and its declared default.""" + + rules_engine = engine + if rules_engine is None: + from microcosm.frame.adapters.policyengine_us import PolicyEngineUSEngine + + rules_engine = PolicyEngineUSEngine() + variables = list(rules_engine.variables()) + defaults = dict(rules_engine.default_values(variables)) + missing_defaults = sorted(set(variables) - set(defaults)) + extra_defaults = sorted(set(defaults) - set(variables)) + if missing_defaults or extra_defaults: + raise ValueError( + "PolicyEngine-US simulation input default surface is not exact; " + f"missing={missing_defaults}, extra={extra_defaults}." + ) + rows = [ + { + "entity": rules_engine.variable_metadata(variable).entity, + "variable": variable, + "default": defaults[variable], + } + for variable in variables + ] + defaults_sha256 = hashlib.sha256( + json.dumps( + rows, + allow_nan=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + ).hexdigest() + observed = { + "engine_version": version("policyengine-us"), + "input_count": len(variables), + "default_count": len(defaults), + "defaults_sha256": defaults_sha256, + } + expected = { + "engine_version": POOL_ENGINE_INPUT_PROJECTION_CONTRACT.engine_version, + "input_count": POOL_ENGINE_INPUT_PROJECTION_CONTRACT.input_count, + "default_count": POOL_ENGINE_INPUT_PROJECTION_CONTRACT.default_count, + "defaults_sha256": (POOL_ENGINE_INPUT_PROJECTION_CONTRACT.defaults_sha256), + } + if observed != expected: + raise ValueError( + "PolicyEngine-US simulation input defaults drifted; refresh the " + f"remaining-stage input audit. expected={expected}, observed={observed}." + ) + return observed + + +@lru_cache(maxsize=8) +def pool_remaining_stage_input_manifest( + metadata_index: PolicyEngineUSVariableMetadataIndex | None = None, +) -> tuple[PoolRemainingStageInput, ...]: + """Enumerate and statically provision every remaining-stage data read. + + The manifest starts at a validated ``transferred`` checkpoint and covers + tail preparation, both derive kernels, all thirteen seed-program branches, + and the terminal SSI simulation. Simulation leaves come from the pinned + source-index graph rather than a handwritten dependency list. + """ + + entries: dict[ + tuple[str, str, str, str], + PoolRemainingStageInput, + ] = {} + + def register( + stage: str, + consumer: str, + entity: str, + variable: str, + *, + execution_scope: str, + provision: str, + available_by: str, + fallback: str | None = None, + ) -> None: + entry = PoolRemainingStageInput( + stage=stage, + consumer=consumer, + entity=entity, + variable=variable, + execution_scope=execution_scope, + provision=provision, + available_by=available_by, + fallback=fallback, + ) + key = (stage, consumer, entity, variable) + previous = entries.setdefault(key, entry) + if previous != entry: + raise ValueError( + "Remaining-stage input has conflicting provisions: " + f"{previous!r} versus {entry!r}." + ) + + surface = {entry.variable: entry for entry in pool_input_surface()} + + def surface_provision(variable: str) -> str: + declaration = surface.get(variable) + if declaration is None: + raise ValueError( + f"Remaining-stage input {variable!r} has no pool input producer." + ) + return f"pool_input_surface:{declaration.family}" + + # The stacked tail step reads only clone provenance and the optional memo + # leaf that it deliberately clears before deterministic re-derivation. + register( + "derive", + "prepare_stacked_tail_derivation", + "person", + support_clone_index_column("person"), + execution_scope="whole_pool", + provision="assembly_support_provenance", + available_by="assembled", + ) + register( + "derive", + "prepare_stacked_tail_derivation", + "person", + "schedule_d_capital_gain_distributions", + execution_scope="clone_2", + provision="optional_existing_derived_leaf", + available_by="transferred", + fallback="absent_or_cleared_then_schedule_d_derived", + ) + + for variable in ( + "long_term_capital_gains_before_response", + "non_sch_d_capital_gains", + ): + register( + "derive", + "_complete_schedule_d_input", + "person", + variable, + execution_scope="whole_pool", + provision=surface_provision(variable), + available_by="transferred", + ) + for entity, variable, provision in ( + ("person", "person_tax_unit_id", "frame_membership"), + ("tax_unit", "tax_unit_id", "frame_entity_id"), + ): + register( + "derive", + "_complete_schedule_d_input", + entity, + variable, + execution_scope="whole_pool", + provision=provision, + available_by="assembled", + ) + register( + "derive", + "_complete_schedule_d_input", + "person", + "schedule_d_capital_gain_distributions", + execution_scope="whole_pool", + provision="optional_transferred_or_schedule_d_derived", + available_by="transferred", + fallback="derive_from_finite_transferred_parents", + ) + + for variable in US_QBI_RECONCILED_PERSON_COLUMNS: + register( + "derive", + "with_us_qbi_input_reconciliation", + "person", + variable, + execution_scope="whole_pool", + provision=surface_provision(variable), + available_by="transferred", + ) + for variable in ( + "partnership_income", + "estate_income", + "non_qualified_dividend_income", + ): + register( + "derive", + "with_us_qbi_input_reconciliation", + "person", + variable, + execution_scope="whole_pool", + provision=surface_provision(variable), + available_by="transferred", + ) + register( + "derive", + "with_us_qbi_input_reconciliation", + "person", + "s_corp_income", + execution_scope="whole_pool", + provision="primary_puf_exact_zero_universe", + available_by="transferred", + ) + for variable, provision in ( + ("age", "assembled_native_person_input"), + ("SEMP", "assembled_raw_acs_source_authority"), + ("person_tax_unit_id", "frame_membership"), + (support_clone_index_column("person"), "assembly_support_provenance"), + ): + register( + "derive", + "with_us_qbi_input_reconciliation", + "person", + variable, + execution_scope="whole_pool", + provision=provision, + available_by="assembled", + ) + universe_owner_inputs = ACS_PUMS_EARNINGS_UNIVERSE_PERSON_INPUTS + for variable in set(universe_owner_inputs) - { + "age", + "person_tax_unit_id", + support_clone_index_column("person"), + support_source_id_column("person"), + }: + register( + "derive", + "with_us_qbi_input_reconciliation", + "person", + variable, + execution_scope="whole_pool", + provision=universe_owner_inputs[variable], + available_by="assembled", + ) + register( + "derive", + "with_us_qbi_input_reconciliation", + "person", + support_source_id_column("person"), + execution_scope="whole_pool", + provision="assembly_support_source_identity", + available_by="assembled", + fallback="person_id_for_unstacked_lineage_digest", + ) + register( + "derive", + "with_us_qbi_input_reconciliation", + "person", + "person_id", + execution_scope="whole_pool", + provision="frame_entity_id", + available_by="assembled", + ) + + contract = load_take_up_contract() + transfer_owned = { + variable + for families in pool_transfer_target_families().values() + for variables in families.values() + for variable in variables + } + for program in contract.programs: + if program.is_seeded: + provision = "administrative_seed_or_preserved_input" + fallback = "sourced_seed_when_input_is_missing" + elif program.variable in transfer_owned: + provision = "transferred_or_preserved_input" + fallback = None + else: + provision = "preserved_input_or_disclosed_engine_default" + fallback = "checked_take_up_contract_engine_default" + register( + "seed", + "seed_multispine_pool_inputs", + program.entity, + program.variable, + execution_scope="whole_pool", + provision=provision, + available_by=( + "transferred" if program.variable in transfer_owned else "seeded" + ), + fallback=fallback, + ) + + # Stable Bernoulli draws consume these structural columns and resolved + # weights. The source-identity triplet is optional as a unit: support or + # entity identity remains the declared deterministic fallback. + for entity in ( + "person", + "household", + "tax_unit", + "spm_unit", + "family", + "marital_unit", + ): + register( + "seed", + "with_us_take_up_inputs", + entity, + support_source_id_column(entity), + execution_scope="whole_pool", + provision="assembly_support_source_identity", + available_by="assembled", + ) + for entity in ("tax_unit", "spm_unit"): + register( + "seed", + "with_us_take_up_inputs", + entity, + f"{entity}_id", + execution_scope="whole_pool", + provision="frame_entity_id", + available_by="assembled", + ) + register( + "seed", + "with_us_take_up_inputs", + "person", + f"person_{entity}_id", + execution_scope="whole_pool", + provision="frame_membership", + available_by="assembled", + ) + register( + "seed", + "with_us_take_up_inputs", + entity, + "", + execution_scope="whole_pool", + provision="frame_resolve_weights_from_household_weight", + available_by="assembled", + ) + for variable in ("source_year", "source_household_id", "source_person_id"): + register( + "seed", + "with_us_take_up_inputs", + "person", + variable, + execution_scope="whole_pool", + provision="optional_assembled_source_identity", + available_by="assembled", + fallback="support_source_id_then_entity_id", + ) + register( + "seed", + "with_us_take_up_inputs", + "person", + "age", + execution_scope="whole_pool", + provision="assembled_native_person_input", + available_by="assembled", + ) + + resolved_metadata_index = ( + metadata_index + if metadata_index is not None + else PolicyEngineUSVariableMetadataIndex() + ) + closure = pool_ssi_dependency_closure(resolved_metadata_index) + take_up_variables = {program.variable for program in contract.programs} + actual_surface_provenance = { + "pool_transfer_target_families", + "PRIMARY_QRF_TARGET_ORDER", + } + ssi_provisions: dict[str, int] = {} + for variable in closure.input_leaves: + metadata = resolved_metadata_index.variable_metadata(variable) + declaration = surface.get(variable) + if variable in POOL_DEFERRED_TRANSFER_INPUTS: + provision = "declared_deferred_null_input" + available_by = "transferred" + fallback = "ephemeral_simulation_projection_engine_default" + elif variable == "age": + provision = "assembled_native_person_input" + available_by = "assembled" + fallback = None + elif variable in take_up_variables: + provision = "seed_stage_program_contract" + available_by = "seeded" + fallback = "seed_receipted_value_or_disclosed_engine_default" + elif declaration is not None and actual_surface_provenance.intersection( + declaration.provenance + ): + provision = "materialized_pool_input_surface" + available_by = "transferred" + fallback = None + else: + provision = "declared_absent_engine_input" + available_by = "simulate" + fallback = "policyengine_default_for_absent_input" + ssi_provisions[provision] = ssi_provisions.get(provision, 0) + 1 + register( + "simulate", + "ssi_static_dependency_closure", + metadata.entity, + variable, + execution_scope="whole_pool", + provision=provision, + available_by=available_by, + fallback=fallback, + ) + + expected_ssi_provisions = { + "assembled_native_person_input": 1, + "materialized_pool_input_surface": 32, + "seed_stage_program_contract": 1, + "declared_deferred_null_input": 3, + "declared_absent_engine_input": 18, + } + if ssi_provisions != expected_ssi_provisions: + raise ValueError( + "SSI input-leaf provisioning drifted; " + f"expected={expected_ssi_provisions}, observed={ssi_provisions}." + ) + + for group in US_SCHEMA.group_entities: + register( + "simulate", + "PolicyEngineUSEngine.materialize", + group, + US_SCHEMA.entity_id_column(group), + execution_scope="whole_pool", + provision="frame_entity_id", + available_by="assembled", + ) + register( + "simulate", + "PolicyEngineUSEngine.materialize", + "person", + US_SCHEMA.membership_column(group), + execution_scope="whole_pool", + provision="frame_membership", + available_by="assembled", + ) + register( + "simulate", + "PolicyEngineUSEngine.materialize", + "person", + US_SCHEMA.person_id_column, + execution_scope="whole_pool", + provision="frame_entity_id", + available_by="assembled", + ) + register( + "simulate", + "PolicyEngineUSEngine.materialize", + "household", + "", + execution_scope="whole_pool", + provision="frame_household_weight", + available_by="assembled", + ) + engine_structural_inputs = { + (group, US_SCHEMA.entity_id_column(group)) for group in US_SCHEMA.group_entities + } | { + ("person", US_SCHEMA.membership_column(group)) + for group in US_SCHEMA.group_entities + } + native_engine_inputs = { + (entity, variable) + for entity, variables in _POOL_NATIVE_COMPLETE_OUTPUTS.items() + for variable in variables + } + projection_provisions: dict[str, int] = {} + for entity, variable in _pool_engine_input_projection( + resolved_metadata_index, + engine_version=closure.engine_version, + ): + fallback: str | None = ( + "ephemeral_simulation_projection_engine_default_if_present_null" + ) + if variable in POOL_DEFERRED_TRANSFER_INPUTS: + provision = "declared_deferred_null_input" + available_by = "transferred" + fallback = "ephemeral_simulation_projection_engine_default" + elif variable in take_up_variables: + provision = "seed_stage_program_contract" + available_by = "seeded" + elif variable in surface: + provision = "materialized_pool_input_surface" + available_by = "transferred" + elif (entity, variable) in native_engine_inputs: + provision = "assembled_native_engine_input" + available_by = "assembled" + elif (entity, variable) in engine_structural_inputs: + provision = "frame_structural_engine_input" + available_by = "assembled" + elif (entity, variable) in _POOL_SIMULATION_PRESERVED_ENGINE_INPUTS: + provision = "preserved_stacked_engine_input" + available_by, preserved_fallback = _POOL_SIMULATION_PRESERVED_ENGINE_INPUTS[ + (entity, variable) + ] + if preserved_fallback is not None: + fallback = preserved_fallback + elif variable == "schedule_d_capital_gain_distributions": + provision = "derived_schedule_d_input" + available_by = "derived" + else: + provision = "declared_absent_engine_input" + available_by = "simulate" + fallback = "policyengine_default_if_absent" + projection_provisions[provision] = projection_provisions.get(provision, 0) + 1 + register( + "simulate", + "_simulation_projection", + entity, + variable, + execution_scope="disposable_simulation_copy", + provision=provision, + available_by=available_by, + fallback=fallback, + ) + + expected_projection_provisions = { + "materialized_pool_input_surface": 123, + "seed_stage_program_contract": 13, + "declared_deferred_null_input": 3, + "assembled_native_engine_input": 5, + "frame_structural_engine_input": 10, + "preserved_stacked_engine_input": 4, + "derived_schedule_d_input": 1, + "declared_absent_engine_input": 704, + } + if projection_provisions != expected_projection_provisions: + raise ValueError( + "Simulation input-projection provisioning drifted; " + f"expected={expected_projection_provisions}, " + f"observed={projection_provisions}." + ) + + return tuple(sorted(entries.values())) + + +def pool_remaining_stage_input_manifest_receipt( + metadata_index: PolicyEngineUSVariableMetadataIndex | None = None, +) -> dict[str, object]: + """Return a content identity for the exhaustive post-transfer manifest.""" + + manifest = pool_remaining_stage_input_manifest(metadata_index) + rows = [ + { + "stage": entry.stage, + "consumer": entry.consumer, + "entity": entry.entity, + "variable": entry.variable, + "execution_scope": entry.execution_scope, + "provision": entry.provision, + "available_by": entry.available_by, + "fallback": entry.fallback, + } + for entry in manifest + ] + manifest_sha256 = hashlib.sha256( + json.dumps( + rows, + allow_nan=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + ).hexdigest() + if manifest_sha256 != POOL_REMAINING_STAGE_INPUT_MANIFEST_SHA256: + raise ValueError( + "Remaining-stage input manifest drifted; refresh its static audit. " + f"expected={POOL_REMAINING_STAGE_INPUT_MANIFEST_SHA256}, " + f"observed={manifest_sha256}." + ) + stage_counts = { + stage: sum(entry.stage == stage for entry in manifest) + for stage in ("derive", "seed", "simulate") + } + consumer_names = sorted({entry.consumer for entry in manifest}) + consumer_counts = { + consumer: sum(entry.consumer == consumer for entry in manifest) + for consumer in consumer_names + } + receipt: dict[str, object] = { + "schema_version": 1, + "entry_count": len(manifest), + "stage_counts": stage_counts, + "consumer_counts": consumer_counts, + "ssi_dependency_contract": { + "engine_version": POOL_SSI_DEPENDENCY_CONTRACT.engine_version, + "root": POOL_SSI_DEPENDENCY_CONTRACT.root, + "input_leaf_count": POOL_SSI_DEPENDENCY_CONTRACT.input_leaf_count, + "formula_node_count": POOL_SSI_DEPENDENCY_CONTRACT.formula_node_count, + "edge_count": POOL_SSI_DEPENDENCY_CONTRACT.edge_count, + "sha256": POOL_SSI_DEPENDENCY_CONTRACT.sha256, + }, + "engine_input_projection_contract": { + "engine_version": POOL_ENGINE_INPUT_PROJECTION_CONTRACT.engine_version, + "input_count": POOL_ENGINE_INPUT_PROJECTION_CONTRACT.input_count, + "default_count": POOL_ENGINE_INPUT_PROJECTION_CONTRACT.default_count, + "sha256": POOL_ENGINE_INPUT_PROJECTION_CONTRACT.sha256, + "defaults_sha256": (POOL_ENGINE_INPUT_PROJECTION_CONTRACT.defaults_sha256), + }, + "manifest_sha256": manifest_sha256, + } + receipt["sha256"] = hashlib.sha256( + json.dumps( + receipt, + allow_nan=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + ).hexdigest() + return receipt + + def materialize_pool_deferred_transfer_inputs(frame: Frame) -> PoolStageOutput: """Represent pool-local source deferrals as typed, all-null input columns. @@ -830,11 +1653,11 @@ def materialize_pool_deferred_transfer_inputs(frame: Frame) -> PoolStageOutput: tables[entity][column] = pd.Series( np.nan, index=tables[entity].index, - dtype=np.float64, + dtype=declaration["physical_dtype"], ) receipts[column] = { **declaration, - "status": "deferred_pending_source_donor", + "status": POOL_DEFERRED_TRANSFER_STATUS, "rows": int(len(tables[entity])), "null_rows": int(len(tables[entity])), } @@ -958,17 +1781,33 @@ def _with_gated_us_hours_worked_inputs(frame: Frame) -> PoolStageOutput: def complete_multispine_source_inputs( frame: Frame, ) -> PoolStageOutput: - """Run clone-safe or clone-required source work after primary imputation. - - The function is intentionally fixed-seed/fixed-period. Each operator runs - over the CPS-evidenced portion of the already assembled and cloned frame; - its declared output family alone is merged into the whole pool. This - retains ACS native measurements, leaves unavailable cells null for the - subsequent declared transfer, and keeps assembly metadata untouched. The - source chain then materializes every pool-local donor deferral as a typed - all-null column with an explicit receipt. + """Run the legacy post-clone source chain through the narrow public API. + + This compatibility entrypoint retains the historical source-only order for + callers whose late inputs are already complete. Production late-stage + orchestration invokes :func:`run_multispine_post_clone_source_operator` + according to the declared producer DAG, then calls + :func:`finalize_multispine_source_inputs` once all sixteen receipts exist. """ + current = frame + operator_receipts: dict[str, Mapping[str, object]] = {} + for operator_name in POOL_POST_CLONE_SOURCE_OPERATOR_ORDER: + completed = run_multispine_post_clone_source_operator( + current, + operator_name, + ) + current = completed.frame + operator_receipts[operator_name] = completed.receipt + return finalize_multispine_source_inputs( + current, + operator_receipts=operator_receipts, + ) + + +def _post_clone_source_operators() -> Mapping[str, SourceFrameOperator]: + """Return the fixed-seed, fixed-period post-clone kernel mapping.""" + operators: Mapping[str, SourceFrameOperator] = { "with_us_prior_year_income_inputs": lambda current: ( with_us_prior_year_income_inputs( @@ -998,42 +1837,51 @@ def complete_multispine_source_inputs( impute_us_housing_assistance_to_puf_support( current, seed=POOL_RANDOM_SEED, + n_estimators=POOL_HOUSING_ASSISTANCE_N_ESTIMATORS, + max_train_samples=POOL_HOUSING_ASSISTANCE_MAX_TRAIN_SAMPLES, ) ), "with_us_child_support_inputs": lambda current: with_us_child_support_inputs( current, seed=POOL_RANDOM_SEED, time_period=POOL_TIME_PERIOD, + allow_existing_without_source=(POOL_SOURCE_ALLOW_EXISTING_WITHOUT_SOURCE), ), "with_us_disability_benefits": lambda current: with_us_disability_benefits( current, seed=POOL_RANDOM_SEED, time_period=POOL_TIME_PERIOD, + allow_existing_without_source=(POOL_SOURCE_ALLOW_EXISTING_WITHOUT_SOURCE), ), "with_us_workers_compensation": lambda current: with_us_workers_compensation( current, seed=POOL_RANDOM_SEED, time_period=POOL_TIME_PERIOD, + allow_existing_without_source=(POOL_SOURCE_ALLOW_EXISTING_WITHOUT_SOURCE), ), "with_us_weeks_unemployed": lambda current: with_us_weeks_unemployed( current, seed=POOL_RANDOM_SEED, time_period=POOL_TIME_PERIOD, + asec_2023_source=None, ), "with_us_childcare_inputs": lambda current: with_us_childcare_inputs( current, seed=POOL_RANDOM_SEED, time_period=POOL_TIME_PERIOD, + allow_existing_without_source=(POOL_SOURCE_ALLOW_EXISTING_WITHOUT_SOURCE), ), "with_us_adult_care_inputs": lambda current: with_us_adult_care_inputs( current, seed=POOL_RANDOM_SEED, time_period=POOL_TIME_PERIOD, + allow_existing_without_source=(POOL_SOURCE_ALLOW_EXISTING_WITHOUT_SOURCE), ), "with_us_energy_subsidy_input": lambda current: with_us_energy_subsidy_input( current, seed=POOL_RANDOM_SEED, time_period=POOL_TIME_PERIOD, + allow_existing_without_source=(POOL_SOURCE_ALLOW_EXISTING_WITHOUT_SOURCE), ), "with_us_retirement_contribution_inputs": lambda current: ( with_us_retirement_contribution_inputs( @@ -1059,20 +1907,143 @@ def complete_multispine_source_inputs( current, seed=POOL_RANDOM_SEED, time_period=POOL_TIME_PERIOD, + asec_education_source=None, ), } - completed = _run_source_operator_chain( + return operators + + +def run_multispine_post_clone_source_operator( + frame: Frame, + operator_name: str, +) -> PoolStageOutput: + """Run exactly one declared post-clone source producer. + + Separating kernel execution from orchestration lets the late-stage DAG + interleave source producers with the transfer groups that fill their + declared inputs. The existing phase, projection, structure, and output + ownership checks remain centralized in the guarded source runner. + """ + + operators = _post_clone_source_operators() + if operator_name not in POOL_POST_CLONE_SOURCE_OPERATOR_ORDER: + raise ValueError( + f"{operator_name!r} is not a declared post-clone source operator; " + f"expected one of {POOL_POST_CLONE_SOURCE_OPERATOR_ORDER}." + ) + if set(operators) != set(POOL_POST_CLONE_SOURCE_OPERATOR_ORDER): + raise RuntimeError( + "Post-clone source operator mapping drifted from its declaration; " + f"missing={sorted(set(POOL_POST_CLONE_SOURCE_OPERATOR_ORDER) - set(operators))}, " + f"unexpected={sorted(set(operators) - set(POOL_POST_CLONE_SOURCE_OPERATOR_ORDER))}." + ) + return _run_source_operator_chain( frame, phase=_POST_CLONE_PHASE, - operator_names=POOL_POST_CLONE_SOURCE_OPERATOR_ORDER, - operators=operators, + operator_names=(operator_name,), + operators={operator_name: operators[operator_name]}, ) - _assert_formula_owned_source_outputs_absent(completed.frame) - deferred = materialize_pool_deferred_transfer_inputs(completed.frame) + + +def finalize_multispine_source_inputs( + frame: Frame, + *, + operator_receipts: Mapping[str, Mapping[str, object]], +) -> PoolStageOutput: + """Validate complete source execution and finalize its persisted surface. + + ``operator_receipts`` must map each of the sixteen declared post-clone + source producers to its single-operator receipt. Mapping insertion order is + retained as the actual execution order, which may be interleaved with + transfer producers by the late-stage DAG. Formula-owned outputs are rejected + before the three explicitly deferred SCF inputs are materialized exactly + once. + """ + + if not isinstance(operator_receipts, Mapping): + raise TypeError("Post-clone source operator receipts must be a mapping.") + receipt_items = tuple(operator_receipts.items()) + normalized: list[dict[str, object]] = [] + operator_order: list[str] = [] + evidence_receipts: list[object] = [] + for receipt_index, (receipt_operator, receipt) in enumerate(receipt_items): + if not isinstance(receipt_operator, str): + raise TypeError( + "Post-clone source operator receipt keys must be strings; " + f"receipt {receipt_index} is keyed by " + f"{type(receipt_operator).__name__}." + ) + if not isinstance(receipt, Mapping): + raise TypeError( + "Post-clone source operator receipts must be mappings; " + f"receipt {receipt_index} is {type(receipt).__name__}." + ) + declared_order = receipt.get("operator_order") + if ( + receipt.get("phase") != _POST_CLONE_PHASE + or not isinstance(declared_order, (list, tuple)) + or len(declared_order) != 1 + or not isinstance(declared_order[0], str) + ): + raise ValueError( + "Each post-clone source receipt must declare phase " + f"{_POST_CLONE_PHASE!r} and exactly one operator; receipt " + f"{receipt_index} was {dict(receipt)!r}." + ) + operator_name = declared_order[0] + if operator_name != receipt_operator: + raise ValueError( + f"Post-clone source receipt key {receipt_operator!r} is " + f"misbound to operator {operator_name!r}." + ) + suboperators = receipt.get("suboperators") + if ( + not isinstance(suboperators, (list, tuple)) + or len(suboperators) != 1 + or not isinstance(suboperators[0], Mapping) + or suboperators[0].get("operator") != operator_name + ): + raise ValueError( + "Each post-clone source receipt must carry the matching single " + f"suboperator receipt for {operator_name!r}." + ) + operator_order.append(operator_name) + normalized_suboperator = dict(suboperators[0]) + normalized_suboperator["order_index"] = receipt_index + normalized.append(normalized_suboperator) + evidence_receipts.append(receipt.get("cps_source_evidence")) + + expected = set(POOL_POST_CLONE_SOURCE_OPERATOR_ORDER) + observed = set(operator_order) + if ( + len(receipt_items) != len(POOL_POST_CLONE_SOURCE_OPERATOR_ORDER) + or observed != expected + ): + raise ValueError( + "Post-clone source finalization requires exactly " + f"{len(POOL_POST_CLONE_SOURCE_OPERATOR_ORDER)} one-operator receipts; " + f"missing={sorted(expected - observed)}, " + f"unexpected={sorted(observed - expected)}." + ) + if evidence_receipts and any( + evidence != evidence_receipts[0] for evidence in evidence_receipts[1:] + ): + raise ValueError( + "Post-clone source receipts disagree on the CPS source-evidence projection." + ) + + _assert_formula_owned_source_outputs_absent(frame) + deferred = materialize_pool_deferred_transfer_inputs(frame) return PoolStageOutput( deferred.frame, { - **completed.receipt, + "phase": _POST_CLONE_PHASE, + "operator_order": operator_order, + "cps_source_evidence": ( + evidence_receipts[0] if evidence_receipts else None + ), + "transient_outputs_carried_through_clone": {}, + "suboperators": normalized, "deferred_transfer_inputs": deferred.receipt, }, ) @@ -1174,6 +2145,46 @@ def _run_source_operator_chain( f"Multispine source operator {operator_name!r} changed entity row " f"counts: input={available_rows}, output={output_rows}." ) + overlap_ownership: Mapping[str, object] | None = None + overlap_targets = ( + set(US_LATE_EDUCATION_NOOP_TARGETS) + if operator_name == "with_us_education_inputs" + else set(US_LATE_RETIREMENT_SOURCE_MIRROR_TARGETS) + if operator_name == "with_us_retirement_contribution_inputs" + else set() + ) + declared_person_outputs = set( + declared_outputs.get(available.schema.person_entity, ()) + ) + available_person_columns = set( + available.table(available.schema.person_entity).columns + ) + overlap_passthrough_required = bool(overlap_targets) and overlap_targets <= ( + available_person_columns + ) + if phase == _POST_CLONE_PHASE and ( + overlap_targets & declared_person_outputs or overlap_passthrough_required + ): + finalized_person, overlap_ownership = _finalize_source_overlap_output( + available.table(available.schema.person_entity), + outcome.table(outcome.schema.person_entity), + operator_name=operator_name, + ) + if overlap_ownership is not None: + tables = {entity: outcome.table(entity) for entity in outcome.entities} + tables.update({link: outcome.link(link) for link in outcome.links}) + tables[outcome.schema.person_entity] = finalized_person + outcome = Frame( + tables, + outcome.schema, + { + entity: outcome.weights_for(entity) + for entity in outcome.weighted_entities + }, + outcome.strata, + mass_log=outcome.mass_log, + metadata=outcome.metadata, + ) _assert_source_operator_structure( available, outcome, @@ -1244,6 +2255,9 @@ def _run_source_operator_chain( }, "formula_owned_outputs_removed": formula_owned_removed, "kernel_receipt": dict(kernel_receipt), + "overlap_ownership": ( + dict(overlap_ownership) if overlap_ownership is not None else None + ), } ) uses_cps_source = any( @@ -1451,6 +2465,164 @@ def _persisted_source_outputs( } +def _numeric_series_byte_receipt( + series: pd.Series, + *, + boundary: str, +) -> dict[str, object]: + values = np.ascontiguousarray(series.to_numpy(copy=False)) + if values.dtype.kind not in "biufc": + raise TypeError( + f"{boundary} requires a physical numeric dtype, got {series.dtype!s}." + ) + digest = hashlib.sha256() + digest.update(str(series.dtype).encode("utf-8")) + digest.update(b"\0") + digest.update(len(series).to_bytes(8, byteorder="little", signed=False)) + digest.update(values.tobytes()) + return { + "dtype": str(series.dtype), + "rows": int(len(series)), + "sha256": digest.hexdigest(), + } + + +def _finalize_source_overlap_output( + before: pd.DataFrame, + after: pd.DataFrame, + *, + operator_name: str, +) -> tuple[pd.DataFrame, dict[str, object] | None]: + """Enforce the reviewed final owner for source-callback overlap cells.""" + + education_operator = "with_us_education_inputs" + retirement_operator = "with_us_retirement_contribution_inputs" + if operator_name not in {education_operator, retirement_operator}: + return after, None + + person_id = "person_id" + required_structure = { + person_id, + support_clone_index_column("person"), + support_source_id_column("person"), + } + missing_structure = sorted( + required_structure - set(before.columns) + | required_structure - set(after.columns) + ) + if missing_structure: + raise ValueError( + f"US late overlap ownership for {operator_name!r} requires " + f"person columns {missing_structure}." + ) + if before[person_id].duplicated().any() or after[person_id].duplicated().any(): + raise ValueError( + f"US late overlap ownership for {operator_name!r} requires unique " + "person_id values." + ) + if set(before[person_id]) != set(after[person_id]): + raise ValueError( + f"US late overlap ownership for {operator_name!r} requires unchanged " + "person_id values." + ) + + result = after.copy(deep=True) + targets_receipt: dict[str, object] = {} + if operator_name == education_operator: + for target in US_LATE_EDUCATION_NOOP_TARGETS: + if target not in before or target not in result: + raise ValueError( + f"US education overlap target person.{target} is absent." + ) + before_values = before.set_index(person_id)[target] + after_values = result.set_index(person_id).loc[before_values.index, target] + before_receipt = _numeric_series_byte_receipt( + before_values, + boundary=f"US education overlap input person.{target}", + ) + after_receipt = _numeric_series_byte_receipt( + after_values, + boundary=f"US education overlap output person.{target}", + ) + if before_receipt != after_receipt: + raise ValueError( + f"US education overlap target person.{target} violated byte " + "identity; its callback is consume-only." + ) + targets_receipt[f"person.{target}"] = { + "action": "consume_only_byte_exact_noop", + "verified_rows": int(len(after_values)), + "byte_identity": after_receipt, + } + else: + clone_column = support_clone_index_column("person") + source_id = support_source_id_column("person") + clone_index = pd.to_numeric(result[clone_column], errors="raise") + clone_values = clone_index.to_numpy(dtype=np.float64) + if not np.equal(clone_values, np.floor(clone_values)).all(): + raise ValueError( + "US retirement overlap ownership requires integral clone roles." + ) + clone_one = clone_index.eq(1) + clone_two = clone_index.eq(2) + parents = result.loc[clone_one] + tails = result.loc[clone_two] + if parents[source_id].duplicated().any() or tails[source_id].duplicated().any(): + raise ValueError( + "US retirement overlap ownership requires unique source IDs " + "within clone roles 1 and 2." + ) + missing_parents = sorted(set(tails[source_id]) - set(parents[source_id])) + if missing_parents: + raise ValueError( + "US retirement overlap ownership found clone-2 rows without " + f"clone-1 parents: {missing_parents}." + ) + parent_by_source = parents.set_index(source_id) + tail_source_ids = tails[source_id] + clone_one_before = result.loc[clone_one].copy(deep=True) + for target in US_LATE_RETIREMENT_SOURCE_MIRROR_TARGETS: + if target not in result: + raise ValueError( + f"US retirement overlap target person.{target} is absent." + ) + expected = parent_by_source.loc[tail_source_ids, target] + expected.index = tails.index + result.loc[clone_two, target] = expected.to_numpy(copy=True) + actual = result.loc[clone_two, target] + expected_receipt = _numeric_series_byte_receipt( + expected, + boundary=f"US retirement overlap parent person.{target}", + ) + actual_receipt = _numeric_series_byte_receipt( + actual, + boundary=f"US retirement overlap tail person.{target}", + ) + if expected_receipt != actual_receipt: + raise ValueError( + f"US retirement overlap target person.{target} failed its " + "byte-exact clone-1 mirror." + ) + targets_receipt[f"person.{target}"] = { + "action": "byte_exact_clone_1_mirror", + "mirrored_clone_2_rows": int(clone_two.sum()), + "byte_identity": actual_receipt, + } + if not result.loc[clone_one].equals(clone_one_before): + raise ValueError( + "US retirement overlap finalization changed source-owned clone-1 " + "rows while mirroring clone 2." + ) + + ownership_receipt = us_late_overlap_ownership_receipt() + return result, { + "passed": True, + "operator": operator_name, + "ownership_sha256": ownership_receipt["sha256"], + "targets": targets_receipt, + } + + def _assert_source_operator_structure( before: Frame, after: Frame, @@ -1553,7 +2725,50 @@ def _merge_source_operator_outputs( f"align one-to-one with the {entity!r} pool." ) for column in sorted(columns): - aligned = source_by_id[column].reindex(target_ids) + source_values = source_by_id[column] + aligned = source_values.reindex(target_ids) + source_is_boolean = _is_physical_boolean_series(source_values) + if source_is_boolean: + positions = np.flatnonzero(eligible.to_numpy()) + aligned_boolean = pd.Series( + pd.array(aligned, dtype="boolean"), + index=target.index, + name=column, + ) + if column not in target: + target[column] = aligned_boolean + continue + incumbent = target[column] + invalid_incumbent = incumbent.dropna().map( + lambda value: not isinstance(value, (bool, np.bool_)) + ) + if invalid_incumbent.any(): + offending_types = sorted( + { + f"{type(value).__module__}.{type(value).__qualname__}" + for value in incumbent.dropna().loc[invalid_incumbent] + } + ) + raise TypeError( + f"Multispine source operator {operator_name!r} emitted " + f"physical booleans for {entity}.{column}, but the pool " + "materialized observed non-boolean values with " + f"dtype {incumbent.dtype!s}: {offending_types}." + ) + merged_boolean = pd.Series( + pd.array(incumbent, dtype="boolean"), + index=target.index, + name=column, + ) + merged_boolean.iloc[positions] = aligned_boolean.iloc[positions].array + target[column] = merged_boolean + continue + if column in target and pd.api.types.is_bool_dtype(target[column].dtype): + raise TypeError( + f"Multispine source operator {operator_name!r} emitted " + f"non-boolean values for boolean-materialized " + f"{entity}.{column}; source dtype={source_values.dtype!s}." + ) if column not in target: target[column] = aligned.to_numpy() else: @@ -1574,6 +2789,18 @@ def _merge_source_operator_outputs( return merged, merged_rows +def _is_physical_boolean_series(values: pd.Series) -> bool: + """Recognize boolean values without treating numeric 0/1 as booleans.""" + + if pd.api.types.is_bool_dtype(values.dtype): + return True + observed = values.dropna() + return bool( + len(observed) + and observed.map(lambda value: isinstance(value, (bool, np.bool_))).all() + ) + + def _frame_row_counts(frame: Frame) -> dict[str, int]: return {entity: int(len(frame.table(entity))) for entity in frame.entities} @@ -1588,6 +2815,8 @@ def derive_multispine_pool_inputs(frame: Frame) -> PoolStageOutput: all-or-nothing identities on the imputed PUF-detail surface. """ + remaining_stage_manifest_receipt = pool_remaining_stage_input_manifest_receipt() + def reconcile_qbi_with_receipt(input_frame: Frame) -> PoolStageOutput: reconciled = with_us_qbi_input_reconciliation(input_frame) receipt = us_qbi_reconciliation_change_receipt(input_frame, reconciled) @@ -1628,6 +2857,7 @@ def reconcile_qbi_with_receipt(input_frame: Frame) -> PoolStageOutput: { "phase": _POST_CLONE_PHASE, "operator_order": list(POOL_DERIVE_OPERATOR_ORDER), + "remaining_stage_input_manifest": remaining_stage_manifest_receipt, "schedule_d_capital_gain_distributions": schedule_d_receipt, "qbi_input_reconciliation": dict(qbi_receipt), }, @@ -1884,6 +3114,11 @@ def materialize_multispine_agreement_outputs( from microcosm.frame.adapters.policyengine_us import PolicyEngineUSEngine rules_engine = PolicyEngineUSEngine() + projection_contract_receipt: Mapping[str, object] = ( + pool_engine_input_projection_receipt(rules_engine) + ) + else: + projection_contract_receipt = {"status": "injected_test_engine"} simulation_frame, default_fills = _simulation_projection(frame, rules_engine) household_ids = simulation_frame.table("household")["household_id"].to_numpy() @@ -1947,6 +3182,7 @@ def materialize_multispine_agreement_outputs( }, "household_batch_size": POOL_SIMULATION_HOUSEHOLD_BATCH_SIZE, "batches": batch_count, + "engine_input_projection_contract": dict(projection_contract_receipt), "simulation_projection_default_fills": default_fills, "persisted_to_pool": False, }, diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/operator_boundary.py b/packages/microcosm-build/src/microcosm/build/us_runtime/operator_boundary.py index b5386fc0..6023ef75 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/operator_boundary.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/operator_boundary.py @@ -32,7 +32,7 @@ US_DISABILITY_BENEFITS_OUTPUT_COLUMNS, ) from microcosm.build.us_runtime.education_inputs import ( - US_EDUCATION_INPUTS_OUTPUT_COLUMNS, + US_EDUCATION_INPUTS_OWNED_OUTPUT_COLUMNS, ) from microcosm.build.us_runtime.eligibility_inputs import ( US_ELIGIBILITY_INPUTS_OUTPUT_COLUMNS, @@ -332,7 +332,7 @@ "person": frozenset(US_ADULT_CARE_OUTPUT_COLUMNS), }, "education_inputs": { - "person": frozenset(US_EDUCATION_INPUTS_OUTPUT_COLUMNS), + "person": frozenset(US_EDUCATION_INPUTS_OWNED_OUTPUT_COLUMNS), }, "scf_wealth": { "person": frozenset(US_SCF_FINANCIAL_ASSET_OUTPUT_COLUMNS), diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_capital_gains_tail.py b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_capital_gains_tail.py index 1c777627..995795bf 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_capital_gains_tail.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_capital_gains_tail.py @@ -5,6 +5,7 @@ import hashlib import json import os +from collections import Counter from collections.abc import Mapping, Sequence from pathlib import Path from typing import Any @@ -19,6 +20,9 @@ ) from microcosm.build.us_runtime.puf_interest_components import ( US_PUF_E19200_AGI_BANDS, + PufE19200AgiBand, + puf_e19200_agi_bands_runtime_identity, + puf_e19200_interest_components_asset_identity, ) from microcosm.build.us_runtime.puf_support import ( PUF_DONOR_SOURCE_ADJUSTED_GROSS_INCOME_COLUMN, @@ -40,25 +44,43 @@ "PUF_CAPITAL_GAINS_TAIL_DONOR_SOURCE_ID_COLUMN", "PUF_CAPITAL_GAINS_TAIL_DONOR_SYNTHETIC_COLUMN", "PUF_CAPITAL_GAINS_TAIL_MANIFEST_SCHEMA_VERSION", + "PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_MAX_TOP_SHARE", + "PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_MIN_NONZERO_RECORDS", + "PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_TOP_K", "PUF_CAPITAL_GAINS_TAIL_PERSON_COLUMNS", "PUF_CAPITAL_GAINS_TAIL_POSITIVE_MASS_FIVE_X_TARGET", "PUF_CAPITAL_GAINS_TAIL_QUANTILE", + "PUF_CAPITAL_GAINS_TAIL_REFERENCE_QUANTILE", + "PUF_CAPITAL_GAINS_TAIL_ASEC_CAPITAL_GAINS_TOPCODE", + "PUF_CAPITAL_GAINS_TAIL_SUPPORT_CONTRACT_VERSION", "PUF_CAPITAL_GAINS_TAIL_STAGE_NAME", "PUF_CAPITAL_GAINS_TAIL_SUPPORT_CHANNEL", "PUF_CAPITAL_GAINS_TAIL_TAX_UNIT_COLUMNS", "PUF_CAPITAL_GAINS_TAIL_TRANSFER_WEIGHT_COLUMN", + "PUF_CAPITAL_GAINS_TAIL_WORSENING_SHARE_TOLERANCE", "assert_puf_capital_gains_tail_survives_selection", "puf_capital_gains_tail_concentration_gate", + "puf_capital_gains_tail_concentration_controls_identity", + "puf_capital_gains_tail_execution_inputs_identity", + "puf_capital_gains_tail_spec_identity", + "puf_capital_gains_tail_support_contract_identity", + "puf_capital_gains_tail_terminal_support_receipt", + "resolve_puf_capital_gains_tail_execution_inputs", "select_puf_capital_gains_tail_donors", "transfer_puf_capital_gains_tail", "validate_puf_capital_gains_tail_manifest", + "validate_puf_capital_gains_tail_terminal_support_receipt", "write_puf_capital_gains_tail_manifest", ] PUF_CAPITAL_GAINS_TAIL_STAGE_NAME = "capital_gains_tail_transfer" PUF_CAPITAL_GAINS_TAIL_SUPPORT_CHANNEL = PUF_TAX_DETAIL_SUPPORT_CHANNEL -PUF_CAPITAL_GAINS_TAIL_MANIFEST_SCHEMA_VERSION = 1 +PUF_CAPITAL_GAINS_TAIL_MANIFEST_SCHEMA_VERSION = 2 +PUF_CAPITAL_GAINS_TAIL_SUPPORT_CONTRACT_VERSION = 1 PUF_CAPITAL_GAINS_TAIL_POSITIVE_MASS_FIVE_X_TARGET = 1_270_900_000_000.0 +PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_TOP_K = 100 +PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_MAX_TOP_SHARE = 0.75 +PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_MIN_NONZERO_RECORDS = 500 # microcosm#567 diagnostic geometry: recipient predictors are bounded by the # $1,999,998 ASEC capital-gains topcode. Weighted q99.5 of positive donor @@ -66,8 +88,8 @@ # above the recipient ceiling. This is a declared source stratum boundary, # not a calibration knob or target multiplier. PUF_CAPITAL_GAINS_TAIL_QUANTILE = 0.995 -_NEXT_REFERENCE_QUANTILE = 0.999 -_ASEC_CAPITAL_GAINS_TOPCODE = 1_999_998.0 +PUF_CAPITAL_GAINS_TAIL_REFERENCE_QUANTILE = 0.999 +PUF_CAPITAL_GAINS_TAIL_ASEC_CAPITAL_GAINS_TOPCODE = 1_999_998.0 PUF_CAPITAL_GAINS_TAIL_PERSON_COLUMNS = ( "short_term_capital_gains", @@ -128,20 +150,151 @@ "short_term_capital_gains", "long_term_capital_gains_before_response", ) -_AGI_UPPER_BOUNDS = np.asarray( - [ - band.upper_bound - for band in US_PUF_E19200_AGI_BANDS - if band.upper_bound is not None - ], - dtype=np.float64, -) + + +def puf_capital_gains_tail_support_contract_identity( + agi_bands: Sequence[PufE19200AgiBand] | None = None, +) -> dict[str, object]: + """Return the immutable per-filing-status recipient-support doctrine.""" + + resolved_agi_bands = tuple( + US_PUF_E19200_AGI_BANDS if agi_bands is None else agi_bands + ) + return { + "contract_id": "puf_capital_gains_tail_per_filing_status_support", + "version": PUF_CAPITAL_GAINS_TAIL_SUPPORT_CONTRACT_VERSION, + "partition": "filing_status", + "filing_statuses": [ + {"filing_status_code": code, "filing_status": name} + for code, name in _FILING_STATUS_BY_CODE.items() + ], + "candidate_universe": ( + "unique single-tax-unit PUF-detail recipient households with " + "weight capacity for the global maximum assigned tail-donor weight" + ), + "required_minimum": "selected_q99_5_tail_donor_count_in_filing_status", + "insufficient_support_action": ( + "skip_entire_filing_status_attachment_without_widening" + ), + "agi_band_policy": ( + "nearest_band_first_then_all_agi_bands_within_filing_status" + ), + "agi_band_count": len(resolved_agi_bands), + } + + +def puf_capital_gains_tail_spec_identity( + spec: PufAggregateDisaggregationSpec | None = None, +) -> dict[str, object]: + """Return the exact resolved aggregate-disaggregation input to tail selection.""" + + resolved = ( + load_default_puf_aggregate_disaggregation_spec() if spec is None else spec + ) + resolved.validate() + return { + "enabled": resolved.enabled, + "forbes_top_tail": resolved.forbes_top_tail, + "source": resolved.source, + "aggregate_recids": list(resolved.aggregate_recids), + "synthetic_recid_start": resolved.synthetic_recid_start, + "screened_fields": list(resolved.screened_fields), + "synthetic_tail_support_eligible": (resolved.synthetic_tail_support_eligible), + "buckets": [ + { + "recid": recid, + "description": bucket.description, + "agi_lower": bucket.agi_lower, + "agi_upper": bucket.agi_upper, + "synthetic_agi_upper": bucket.synthetic_agi_upper, + } + for recid, bucket in sorted(resolved.buckets.items()) + ], + } + + +def puf_capital_gains_tail_concentration_controls_identity() -> dict[str, object]: + """Return the explicit selected-tail and produced-frame gate controls.""" + + return { + "schema_version": 2, + "selection_quantile": PUF_CAPITAL_GAINS_TAIL_QUANTILE, + "selection_comparison": "strictly_greater_than", + "reference_quantile": PUF_CAPITAL_GAINS_TAIL_REFERENCE_QUANTILE, + "recipient_capital_gains_topcode": ( + PUF_CAPITAL_GAINS_TAIL_ASEC_CAPITAL_GAINS_TOPCODE + ), + "positive_mass_five_x_target": ( + PUF_CAPITAL_GAINS_TAIL_POSITIVE_MASS_FIVE_X_TARGET + ), + "worsening_share_tolerance": (PUF_CAPITAL_GAINS_TAIL_WORSENING_SHARE_TOLERANCE), + "ordered_recipient_agi_proxy_columns": list(_RECIPIENT_AGI_PROXY_COLUMNS), + "ordered_joint_vector_columns": list(_JOINT_VECTOR_COLUMNS), + "recipient_owned_candidate_overlap": sorted(_RECIPIENT_OWNED_CANDIDATE_OVERLAP), + "top_k": PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_TOP_K, + "max_top_share": PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_MAX_TOP_SHARE, + "min_nonzero_records": ( + PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_MIN_NONZERO_RECORDS + ), + "reviewed_exclusions": {}, + } + + +def resolve_puf_capital_gains_tail_execution_inputs() -> tuple[ + PufAggregateDisaggregationSpec, + tuple[PufE19200AgiBand, ...], +]: + """Resolve once the exact spec and SOI bands consumed by one tail run.""" + + spec = load_default_puf_aggregate_disaggregation_spec() + spec.validate() + return spec, tuple(US_PUF_E19200_AGI_BANDS) + + +def puf_capital_gains_tail_execution_inputs_identity( + *, + spec: PufAggregateDisaggregationSpec | None = None, + agi_bands: Sequence[PufE19200AgiBand] | None = None, +) -> dict[str, object]: + """Bind the data assets and controls read by the tail callback.""" + + if spec is None or agi_bands is None: + default_spec, default_agi_bands = ( + resolve_puf_capital_gains_tail_execution_inputs() + ) + resolved_spec = default_spec if spec is None else spec + resolved_agi_bands = ( + default_agi_bands if agi_bands is None else tuple(agi_bands) + ) + else: + resolved_spec = spec + resolved_agi_bands = tuple(agi_bands) + soi_asset = puf_e19200_interest_components_asset_identity() + runtime_agi_bands = puf_e19200_agi_bands_runtime_identity(resolved_agi_bands) + if runtime_agi_bands["agi_bands"] != soi_asset["agi_bands"]: + raise ValueError( + "PUF capital-gains tail runtime SOI AGI bands differ from the " + "content-bound packaged asset." + ) + return { + "aggregate_disaggregation_spec": puf_capital_gains_tail_spec_identity( + resolved_spec + ), + "soi_e19200_agi_bands": { + **soi_asset, + "runtime_agi_bands": runtime_agi_bands, + }, + "concentration_gate": ( + puf_capital_gains_tail_concentration_controls_identity() + ), + } def select_puf_capital_gains_tail_donors( donor: pd.DataFrame, *, spec: PufAggregateDisaggregationSpec | None = None, + agi_bands: Sequence[PufE19200AgiBand] | None = None, ) -> tuple[pd.DataFrame, dict[str, object]]: """Select the declared weighted-positive ST+LT tail stratum.""" @@ -155,8 +308,13 @@ def select_puf_capital_gains_tail_donors( missing = sorted(required - set(donor.columns)) if missing: raise ValueError(f"PUF capital-gains tail donor missing columns: {missing}.") - resolved_spec = spec or load_default_puf_aggregate_disaggregation_spec() + resolved_spec = ( + load_default_puf_aggregate_disaggregation_spec() if spec is None else spec + ) resolved_spec.validate() + resolved_agi_bands = tuple( + US_PUF_E19200_AGI_BANDS if agi_bands is None else agi_bands + ) numeric = donor.copy() for column in required: @@ -196,18 +354,20 @@ def select_puf_capital_gains_tail_donors( next_reference_boundary = _weighted_quantile( combined[positive], weights[positive], - _NEXT_REFERENCE_QUANTILE, + PUF_CAPITAL_GAINS_TAIL_REFERENCE_QUANTILE, ) - if boundary > _ASEC_CAPITAL_GAINS_TOPCODE: + if boundary > PUF_CAPITAL_GAINS_TAIL_ASEC_CAPITAL_GAINS_TOPCODE: raise ValueError( "PUF capital-gains q99.5 boundary exceeds the measured ASEC " - f"recipient topcode: {boundary} > {_ASEC_CAPITAL_GAINS_TOPCODE}." + "recipient topcode: " + f"{boundary} > {PUF_CAPITAL_GAINS_TAIL_ASEC_CAPITAL_GAINS_TOPCODE}." ) - if next_reference_boundary <= _ASEC_CAPITAL_GAINS_TOPCODE: + if next_reference_boundary <= PUF_CAPITAL_GAINS_TAIL_ASEC_CAPITAL_GAINS_TOPCODE: raise ValueError( "PUF capital-gains diagnostic geometry changed: weighted-positive " - f"q{_NEXT_REFERENCE_QUANTILE}={next_reference_boundary} no longer " - f"exceeds the ASEC recipient topcode {_ASEC_CAPITAL_GAINS_TOPCODE}." + f"q{PUF_CAPITAL_GAINS_TAIL_REFERENCE_QUANTILE}=" + f"{next_reference_boundary} no longer exceeds the ASEC recipient " + f"topcode {PUF_CAPITAL_GAINS_TAIL_ASEC_CAPITAL_GAINS_TOPCODE}." ) # The stratum is strictly above the declared boundary. This follows the @@ -225,11 +385,12 @@ def select_puf_capital_gains_tail_donors( tail[PUF_DONOR_SOURCE_ADJUSTED_GROSS_INCOME_COLUMN].to_numpy( dtype=np.float64, copy=False, - ) + ), + agi_bands=resolved_agi_bands, ) tail[_TAIL_AGI_BAND_INDEX_COLUMN] = band_index tail[_TAIL_AGI_BAND_LABEL_COLUMN] = [ - US_PUF_E19200_AGI_BANDS[index].label for index in band_index + resolved_agi_bands[index].label for index in band_index ] tail.sort_values("tax_unit_id", kind="mergesort", inplace=True) tail.reset_index(drop=True, inplace=True) @@ -247,9 +408,9 @@ def select_puf_capital_gains_tail_donors( "quantile": PUF_CAPITAL_GAINS_TAIL_QUANTILE, "comparison": "strictly_greater_than", "realized_boundary": float(boundary), - "next_reference_quantile": _NEXT_REFERENCE_QUANTILE, + "next_reference_quantile": PUF_CAPITAL_GAINS_TAIL_REFERENCE_QUANTILE, "next_reference_boundary": float(next_reference_boundary), - "recipient_topcode": _ASEC_CAPITAL_GAINS_TOPCODE, + "recipient_topcode": PUF_CAPITAL_GAINS_TAIL_ASEC_CAPITAL_GAINS_TOPCODE, "eligible_positive_record_count": int(positive.sum()), "eligible_positive_weight": float(weights[positive].sum()), "eligible_positive_mass": eligible_positive_mass, @@ -294,6 +455,10 @@ def puf_capital_gains_tail_concentration_gate( return tail_concentration_gate( values, {column: resolved_weights for column in values}, + top_k=PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_TOP_K, + max_top_share=PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_MAX_TOP_SHARE, + min_nonzero_records=(PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_MIN_NONZERO_RECORDS), + reviewed_exclusions={}, ) @@ -393,14 +558,16 @@ def tail_rows(frame: Frame) -> pd.DataFrame: #: weight splitting and changed summation order move shares by ULPs; a #: strict > would fail 0.84 -> 0.8400000000000001. Anything below this is #: numerical noise, not a worsening. -_WORSENING_SHARE_TOLERANCE = 1e-9 +PUF_CAPITAL_GAINS_TAIL_WORSENING_SHARE_TOLERANCE = 1e-9 +# Retained for the low-level comparator regression and private callers. +_WORSENING_SHARE_TOLERANCE = PUF_CAPITAL_GAINS_TAIL_WORSENING_SHARE_TOLERANCE def _raw_top_share_receipts( values_by_column: Mapping[str, np.ndarray], weights: np.ndarray, *, - top_k: int = 100, + top_k: int | None = None, ) -> dict[str, dict[str, object]]: """Measure every column's weighted top-share raw — no thin-column skip. @@ -411,6 +578,9 @@ def _raw_top_share_receipts( as a worsening from zero. The comparator must see the raw geometry. """ + resolved_top_k = ( + PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_TOP_K if top_k is None else top_k + ) weight_vector = np.asarray(weights, dtype=np.float64) receipts: dict[str, dict[str, object]] = {} for column, values in values_by_column.items(): @@ -427,7 +597,7 @@ def _raw_top_share_receipts( carriers = int(masked_mass.size) total = float(masked_mass.sum()) if total > 0.0: - top = np.sort(masked_mass)[::-1][:top_k] + top = np.sort(masked_mass)[::-1][:resolved_top_k] share = float(top.sum() / total) else: share = 0.0 @@ -467,7 +637,9 @@ def _stage_attributable_concentration_failures( pre_share = float(pre.get("top_share", 0.0)) post_share = float(post_receipts[column]["top_share"]) over = column in over_threshold - worsened = post_share > pre_share + _WORSENING_SHARE_TOLERANCE + worsened = ( + post_share > pre_share + PUF_CAPITAL_GAINS_TAIL_WORSENING_SHARE_TOLERANCE + ) receipts[column] = { "pre_stage_top_share": pre_share, "post_stage_top_share": post_share, @@ -493,6 +665,7 @@ def transfer_puf_capital_gains_tail( *, seed: int, spec: PufAggregateDisaggregationSpec | None = None, + agi_bands: Sequence[PufE19200AgiBand] | None = None, ) -> tuple[Frame, dict[str, object]]: """Split PUF households and transfer exact joint donor-tail vectors.""" @@ -510,25 +683,31 @@ def transfer_puf_capital_gains_tail( if not isinstance(seed, (int, np.integer)) or int(seed) < 0: raise ValueError("PUF capital-gains tail seed must be a nonnegative integer.") - resolved_spec = spec or load_default_puf_aggregate_disaggregation_spec() - tail, selection = select_puf_capital_gains_tail_donors( + resolved_spec = ( + load_default_puf_aggregate_disaggregation_spec() if spec is None else spec + ) + resolved_agi_bands = tuple( + US_PUF_E19200_AGI_BANDS if agi_bands is None else agi_bands + ) + selected_tail, selection = select_puf_capital_gains_tail_donors( donor, spec=resolved_spec, + agi_bands=resolved_agi_bands, ) donor_weight_total = float(pd.to_numeric(donor["weight"], errors="raise").sum()) if not np.isfinite(donor_weight_total) or donor_weight_total <= 0.0: raise ValueError("PUF donor total weight must be positive and finite.") frame_household_weight_total = frame.weights_for("household").total design_weight_normalization = frame_household_weight_total / donor_weight_total - assigned_weights = ( - tail["weight"].to_numpy(dtype=np.float64) * design_weight_normalization + selected_assigned_weights = ( + selected_tail["weight"].to_numpy(dtype=np.float64) * design_weight_normalization ) - if not (assigned_weights > 0.0).all(): + if not (selected_assigned_weights > 0.0).all(): raise ValueError("Every selected PUF tail donor must receive positive mass.") concentration = puf_capital_gains_tail_concentration_gate( - tail, - weights=assigned_weights, + selected_tail, + weights=selected_assigned_weights, ) if not concentration.passed: raise ValueError( @@ -538,13 +717,40 @@ def transfer_puf_capital_gains_tail( candidates = _recipient_candidates( frame, - maximum_transfer_weight=float(assigned_weights.max()), + maximum_transfer_weight=float(selected_assigned_weights.max()), seed=int(seed), + agi_bands=resolved_agi_bands, + ) + recipient_support = _recipient_support_receipt( + selected_tail, + candidates, + agi_bands=resolved_agi_bands, ) + attached_codes = { + int(stratum["filing_status_code"]) + for stratum in recipient_support["strata"] + if stratum["status"] == "attached" + } + attached_mask = ( + pd.to_numeric(selected_tail["filing_status_code"], errors="raise") + .astype("int64") + .isin(attached_codes) + .to_numpy() + ) + if not attached_mask.any(): + insufficient = recipient_support["insufficient_support_strata"] + raise ValueError( + "PUF capital-gains tail no_attachable_strata: every selected " + "filing-status stratum has insufficient support; " + f"insufficient_support={insufficient}." + ) + tail = selected_tail.loc[attached_mask].reset_index(drop=True) + assigned_weights = selected_assigned_weights[attached_mask] assignments = _assign_tail_donors( tail, assigned_weights=assigned_weights, candidates=candidates, + agi_bands=resolved_agi_bands, ) # Fidelity is asserted by construction (microcosm#570 review): every # joint-vector column in every assignment must equal the SELECTED @@ -621,6 +827,10 @@ def transfer_puf_capital_gains_tail( tail[_TAIL_COMBINED_COLUMN].to_numpy(dtype=np.float64), tail["weight"].to_numpy(dtype=np.float64), ) + selected_donor_tail_distribution = _distribution_receipt( + selected_tail[_TAIL_COMBINED_COLUMN].to_numpy(dtype=np.float64), + selected_tail["weight"].to_numpy(dtype=np.float64), + ) signed_reconciliation: dict[str, dict[str, float]] = {} for column in ( "short_term_capital_gains", @@ -678,7 +888,12 @@ def transfer_puf_capital_gains_tail( "frame_household_weight_total": frame_household_weight_total, "design_weight_normalization": float(design_weight_normalization), "assigned_tail_weight": float(assigned_weights.sum()), + "selected_tail_weight": float(selected_assigned_weights.sum()), + "skipped_tail_weight": float( + selected_assigned_weights.sum() - assigned_weights.sum() + ), }, + "recipient_support": recipient_support, "joint_vector_columns": list(_JOINT_VECTOR_COLUMNS), "joint_vector_policy": { "amount_scale": 1.0, @@ -691,6 +906,7 @@ def transfer_puf_capital_gains_tail( "carrier_reconciliation": carrier_reconciliation, "tail_distribution_receipts": { "donor": donor_tail_distribution, + "selected_donor": selected_donor_tail_distribution, "frame_transferred": frame_tail_distribution, "frame_before_stage": before_distribution, "frame_after_stage": after_distribution, @@ -737,16 +953,289 @@ def write_puf_capital_gains_tail_manifest( return hashlib.sha256(output.read_bytes()).hexdigest() +def _validate_recipient_support_receipt( + receipt: object, + *, + records: Sequence[Mapping[str, object]] | None, + selected_donor_count: int | None, +) -> None: + if not isinstance(receipt, Mapping): + raise ValueError( + "PUF capital-gains tail manifest recipient support must be an object." + ) + support = dict(receipt) + claimed = support.pop("sha256", None) + actual = _canonical_sha256(support) + if claimed != actual: + raise ValueError( + "PUF capital-gains tail recipient-support SHA mismatch: " + f"claimed {claimed!r}, computed {actual!r}." + ) + expected_fields = { + "contract", + "candidate_count", + "selected_donor_count", + "attached_donor_count", + "skipped_donor_count", + "attached_stratum_count", + "insufficient_support_stratum_count", + "not_applicable_stratum_count", + "insufficient_support_strata", + "strata", + } + if set(support) != expected_fields: + raise ValueError( + "PUF capital-gains tail recipient-support receipt schema mismatch." + ) + if support["contract"] != puf_capital_gains_tail_support_contract_identity(): + raise ValueError( + "PUF capital-gains tail recipient-support contract identity changed." + ) + + def nonnegative_integer(field: str) -> int: + value = support.get(field) + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise ValueError( + "PUF capital-gains tail recipient-support " + f"{field} must be a nonnegative integer." + ) + return value + + strata = support.get("strata") + if not isinstance(strata, list) or len(strata) != len(_FILING_STATUS_BY_CODE): + raise ValueError( + "PUF capital-gains tail recipient-support strata must enumerate " + "all five filing statuses exactly once." + ) + expected_stratum_fields = { + "filing_status_code", + "filing_status", + "status", + "observed_count", + "required_minimum", + "attached_donor_count", + "skipped_donor_count", + } + required_total = 0 + observed_total = 0 + attached_total = 0 + skipped_total = 0 + attached_strata = 0 + insufficient: list[str] = [] + not_applicable = 0 + attached_by_code: dict[int, int] = {} + for (expected_code, expected_name), raw_stratum in zip( + _FILING_STATUS_BY_CODE.items(), + strata, + strict=True, + ): + if not isinstance(raw_stratum, Mapping) or set(raw_stratum) != ( + expected_stratum_fields + ): + raise ValueError( + "PUF capital-gains tail recipient-support stratum schema mismatch." + ) + stratum = dict(raw_stratum) + if ( + stratum["filing_status_code"] != expected_code + or stratum["filing_status"] != expected_name + ): + raise ValueError( + "PUF capital-gains tail recipient-support filing-status order " + "or identity changed." + ) + counts: dict[str, int] = {} + for field in ( + "observed_count", + "required_minimum", + "attached_donor_count", + "skipped_donor_count", + ): + value = stratum[field] + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise ValueError( + "PUF capital-gains tail recipient-support stratum counts " + "must be nonnegative integers." + ) + counts[field] = value + observed = counts["observed_count"] + required = counts["required_minimum"] + attached = counts["attached_donor_count"] + skipped = counts["skipped_donor_count"] + status = stratum["status"] + if required == 0: + valid = status == "not_applicable" and attached == skipped == 0 + not_applicable += 1 + elif observed < required: + valid = ( + status == "insufficient_support" + and attached == 0 + and skipped == required + ) + insufficient.append(expected_name) + else: + valid = status == "attached" and attached == required and skipped == 0 + attached_strata += 1 + if not valid: + raise ValueError( + "PUF capital-gains tail recipient-support status/count " + f"arithmetic is inconsistent for {expected_name}." + ) + required_total += required + observed_total += observed + attached_total += attached + skipped_total += skipped + attached_by_code[expected_code] = attached + + if nonnegative_integer("candidate_count") != observed_total: + raise ValueError( + "PUF capital-gains tail recipient-support candidate count does not " + "equal its strata." + ) + receipt_selected = nonnegative_integer("selected_donor_count") + if receipt_selected != required_total or ( + selected_donor_count is not None and receipt_selected != selected_donor_count + ): + raise ValueError( + "PUF capital-gains tail recipient-support selected donor count does " + "not equal its declared requirement." + ) + expected_summary = { + "attached_donor_count": attached_total, + "skipped_donor_count": skipped_total, + "attached_stratum_count": attached_strata, + "insufficient_support_stratum_count": len(insufficient), + "not_applicable_stratum_count": not_applicable, + } + for field, expected in expected_summary.items(): + if nonnegative_integer(field) != expected: + raise ValueError( + "PUF capital-gains tail recipient-support summary arithmetic " + f"is inconsistent for {field}." + ) + if support.get("insufficient_support_strata") != insufficient: + raise ValueError( + "PUF capital-gains tail insufficient-support stratum names changed." + ) + if records is not None: + record_counts: Counter[int] = Counter() + for record in records: + if not isinstance(record, Mapping): + raise ValueError( + "PUF capital-gains tail manifest records must contain objects." + ) + try: + code = int(record["donor_filing_status_code"]) + name = str(record["donor_filing_status"]) + except (KeyError, TypeError, ValueError) as error: + raise ValueError( + "PUF capital-gains tail record filing status is malformed." + ) from error + if _FILING_STATUS_BY_CODE.get(code) != name: + raise ValueError( + "PUF capital-gains tail record filing-status identity changed." + ) + record_counts[code] += 1 + if dict(record_counts) != { + code: count for code, count in attached_by_code.items() if count + }: + raise ValueError( + "PUF capital-gains tail attached records do not equal the " + "recipient-support status counts." + ) + + +def puf_capital_gains_tail_terminal_support_receipt( + manifest: Mapping[str, object], +) -> dict[str, object]: + """Project one validated tail-support receipt into sealed terminal gates.""" + + validate_puf_capital_gains_tail_manifest(manifest) + payload: dict[str, object] = { + "artifact_kind": "populace_puf_capital_gains_tail_terminal_support", + "tail_manifest_schema_version": PUF_CAPITAL_GAINS_TAIL_MANIFEST_SCHEMA_VERSION, + "tail_manifest_sha256": manifest["manifest_sha256"], + "recipient_support": json.loads( + json.dumps(manifest["recipient_support"], allow_nan=False) + ), + } + payload["sha256"] = _canonical_sha256(payload) + return payload + + +def validate_puf_capital_gains_tail_terminal_support_receipt( + receipt: Mapping[str, object], +) -> str: + """Fail closed on a mutated terminal projection of tail support.""" + + if not isinstance(receipt, Mapping): + raise ValueError("PUF capital-gains tail terminal support must be an object.") + payload = dict(receipt) + claimed = payload.pop("sha256", None) + if set(payload) != { + "artifact_kind", + "tail_manifest_schema_version", + "tail_manifest_sha256", + "recipient_support", + }: + raise ValueError("PUF capital-gains tail terminal support schema mismatch.") + if ( + payload["artifact_kind"] != "populace_puf_capital_gains_tail_terminal_support" + or payload["tail_manifest_schema_version"] + != PUF_CAPITAL_GAINS_TAIL_MANIFEST_SCHEMA_VERSION + ): + raise ValueError("PUF capital-gains tail terminal support identity changed.") + tail_sha = payload["tail_manifest_sha256"] + if ( + not isinstance(tail_sha, str) + or len(tail_sha) != 64 + or any(character not in "0123456789abcdef" for character in tail_sha) + ): + raise ValueError( + "PUF capital-gains tail terminal support manifest SHA is malformed." + ) + _validate_recipient_support_receipt( + payload["recipient_support"], + records=None, + selected_donor_count=None, + ) + actual = _canonical_sha256(payload) + if claimed != actual: + raise ValueError( + "PUF capital-gains tail terminal-support SHA mismatch: " + f"claimed {claimed!r}, computed {actual!r}." + ) + return actual + + def validate_puf_capital_gains_tail_manifest( manifest: Mapping[str, object], ) -> str: """Validate all manifest hashes and return the canonical payload SHA-256.""" + if not isinstance(manifest, Mapping): + raise ValueError("PUF capital-gains tail manifest must be an object.") payload = dict(manifest) claimed = payload.pop("manifest_sha256", None) + if ( + payload.get("artifact_kind") != "populace_puf_capital_gains_tail_transfer" + or payload.get("schema_version") + != PUF_CAPITAL_GAINS_TAIL_MANIFEST_SCHEMA_VERSION + or payload.get("stage") != PUF_CAPITAL_GAINS_TAIL_STAGE_NAME + ): + raise ValueError( + "PUF capital-gains tail manifest artifact/schema/stage identity changed." + ) records = payload.get("records") if not isinstance(records, list): raise ValueError("PUF capital-gains tail manifest records must be a list.") + record_count = payload.get("record_count") + if ( + isinstance(record_count, bool) + or not isinstance(record_count, int) + or record_count != len(records) + ): + raise ValueError("PUF capital-gains tail manifest record count changed.") donor_records_sha256 = _canonical_sha256(_donor_record_projection(records)) if payload.get("donor_records_sha256") != donor_records_sha256: raise ValueError( @@ -761,6 +1250,23 @@ def validate_puf_capital_gains_tail_manifest( f"claimed {payload.get('assignment_sha256')!r}, " f"computed {assignment_sha256!r}." ) + boundary = payload.get("boundary") + if not isinstance(boundary, Mapping): + raise ValueError("PUF capital-gains tail manifest boundary is malformed.") + selected_donor_count = boundary.get("tail_record_count") + if ( + isinstance(selected_donor_count, bool) + or not isinstance(selected_donor_count, int) + or selected_donor_count <= 0 + ): + raise ValueError( + "PUF capital-gains tail manifest selected donor count is malformed." + ) + _validate_recipient_support_receipt( + payload.get("recipient_support"), + records=records, + selected_donor_count=selected_donor_count, + ) actual = _canonical_sha256(payload) if claimed != actual: raise ValueError( @@ -775,7 +1281,11 @@ def _recipient_candidates( *, maximum_transfer_weight: float, seed: int, + agi_bands: Sequence[PufE19200AgiBand] | None = None, ) -> pd.DataFrame: + resolved_agi_bands = tuple( + US_PUF_E19200_AGI_BANDS if agi_bands is None else agi_bands + ) person = frame.table("person") household = frame.table("household") tax_unit = frame.table("tax_unit") @@ -880,10 +1390,11 @@ def _recipient_candidates( ) puf_tax_units["recipient_agi_proxy"] = proxy_values.to_numpy(dtype=np.float64) puf_tax_units["recipient_agi_band_index"] = _agi_band_indices( - puf_tax_units["recipient_agi_proxy"].to_numpy(dtype=np.float64) + puf_tax_units["recipient_agi_proxy"].to_numpy(dtype=np.float64), + agi_bands=resolved_agi_bands, ) puf_tax_units["recipient_agi_band"] = [ - US_PUF_E19200_AGI_BANDS[index].label + resolved_agi_bands[index].label for index in puf_tax_units["recipient_agi_band_index"] ] puf_tax_units["recipient_household_source_id"] = puf_tax_units[ @@ -907,12 +1418,103 @@ def _recipient_candidates( return puf_tax_units.reset_index(drop=True) +def _recipient_support_receipt( + tail: pd.DataFrame, + candidates: pd.DataFrame, + *, + agi_bands: Sequence[PufE19200AgiBand] | None = None, +) -> dict[str, object]: + """Count the declared support universe before any status is attached.""" + + donor_codes = pd.to_numeric(tail["filing_status_code"], errors="raise").astype( + "int64" + ) + candidate_codes = pd.to_numeric( + candidates["recipient_filing_status_code"], + errors="raise", + ).astype("int64") + unknown_donor_codes = sorted(set(donor_codes) - set(_FILING_STATUS_BY_CODE)) + unknown_candidate_codes = sorted(set(candidate_codes) - set(_FILING_STATUS_BY_CODE)) + if unknown_donor_codes or unknown_candidate_codes: + raise ValueError( + "PUF capital-gains tail support audit found unknown filing-status " + f"codes: donors={unknown_donor_codes}, " + f"candidates={unknown_candidate_codes}." + ) + if candidates["recipient_household_id"].duplicated().any(): + raise ValueError( + "PUF capital-gains tail support candidates must be unique households." + ) + + required_counts = donor_codes.value_counts().to_dict() + observed_counts = candidate_codes.value_counts().to_dict() + strata: list[dict[str, object]] = [] + for code, name in _FILING_STATUS_BY_CODE.items(): + required = int(required_counts.get(code, 0)) + observed = int(observed_counts.get(code, 0)) + if required == 0: + status = "not_applicable" + attached = 0 + skipped = 0 + elif observed < required: + status = "insufficient_support" + attached = 0 + skipped = required + else: + status = "attached" + attached = required + skipped = 0 + strata.append( + { + "filing_status_code": code, + "filing_status": name, + "status": status, + "observed_count": observed, + "required_minimum": required, + "attached_donor_count": attached, + "skipped_donor_count": skipped, + } + ) + + insufficient = [ + str(stratum["filing_status"]) + for stratum in strata + if stratum["status"] == "insufficient_support" + ] + payload: dict[str, object] = { + "contract": puf_capital_gains_tail_support_contract_identity(agi_bands), + "candidate_count": int(len(candidates)), + "selected_donor_count": int(len(tail)), + "attached_donor_count": int( + sum(int(stratum["attached_donor_count"]) for stratum in strata) + ), + "skipped_donor_count": int( + sum(int(stratum["skipped_donor_count"]) for stratum in strata) + ), + "attached_stratum_count": int( + sum(stratum["status"] == "attached" for stratum in strata) + ), + "insufficient_support_stratum_count": len(insufficient), + "not_applicable_stratum_count": int( + sum(stratum["status"] == "not_applicable" for stratum in strata) + ), + "insufficient_support_strata": insufficient, + "strata": strata, + } + payload["sha256"] = _canonical_sha256(payload) + return payload + + def _assign_tail_donors( tail: pd.DataFrame, *, assigned_weights: np.ndarray, candidates: pd.DataFrame, + agi_bands: Sequence[PufE19200AgiBand] | None = None, ) -> pd.DataFrame: + resolved_agi_bands = tuple( + US_PUF_E19200_AGI_BANDS if agi_bands is None else agi_bands + ) if len(tail) != len(assigned_weights): raise ValueError("Assigned tail weights must align with donor rows.") queues: dict[tuple[int, int], list[int]] = {} @@ -945,7 +1547,7 @@ def _assign_tail_donors( donor_band = int(donor_row[_TAIL_AGI_BAND_INDEX_COLUMN]) candidate_index: int | None = None for band in sorted( - range(len(US_PUF_E19200_AGI_BANDS)), + range(len(resolved_agi_bands)), key=lambda value: (abs(value - donor_band), value), ): queue = queues.get((filing_status_code, band)) @@ -1368,6 +1970,10 @@ def _frame_capital_gains_concentration_gate(frame: Frame) -> GateResult: return tail_concentration_gate( values, {column: weights for column in values}, + top_k=PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_TOP_K, + max_top_share=PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_MAX_TOP_SHARE, + min_nonzero_records=(PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_MIN_NONZERO_RECORDS), + reviewed_exclusions={}, ) @@ -1750,11 +2356,19 @@ def _assignment_record_projection( return [{key: record[key] for key in keys} for record in records] -def _agi_band_indices(values: Sequence[float]) -> np.ndarray: +def _agi_band_indices( + values: Sequence[float], + *, + agi_bands: Sequence[PufE19200AgiBand] = US_PUF_E19200_AGI_BANDS, +) -> np.ndarray: numeric = np.asarray(values, dtype=np.float64) if numeric.ndim != 1 or not np.isfinite(numeric).all(): raise ValueError("PUF tail AGI values must be one-dimensional and finite.") - return np.searchsorted(_AGI_UPPER_BOUNDS, numeric, side="right") + upper_bounds = np.asarray( + [band.upper_bound for band in agi_bands if band.upper_bound is not None], + dtype=np.float64, + ) + return np.searchsorted(upper_bounds, numeric, side="right") def _weighted_quantile( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_interest_components.py b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_interest_components.py index 38ec8ee1..5a22f3f5 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_interest_components.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_interest_components.py @@ -2,7 +2,9 @@ from __future__ import annotations +import hashlib import json +from collections.abc import Sequence from dataclasses import dataclass from importlib.resources import files from typing import Any @@ -95,11 +97,18 @@ def _band(raw: dict[str, Any]) -> PufE19200AgiBand: ) -def _load_source_asset() -> tuple[ +def _load_source_asset( + resource: Any | None = None, +) -> tuple[ PufE19200InterestComponents, tuple[PufE19200AgiBand, ...], ]: - payload = json.loads(files("microcosm.build.us").joinpath(_SOURCE_ASSET).read_text()) + resolved_resource = ( + files("microcosm.build.us").joinpath(_SOURCE_ASSET) + if resource is None + else resource + ) + payload = json.loads(resolved_resource.read_text(encoding="utf-8")) source = payload.get("source", {}) if ( source.get("tax_year") != 2015 @@ -142,6 +151,54 @@ def _load_source_asset() -> tuple[ return all_returns, bands +def _component_identity( + row: PufE19200InterestComponents, +) -> dict[str, object]: + return { + "source_row": row.source_row, + "total_interest_paid_amount": row.total_interest_paid_amount, + "home_mortgage_interest_amount": row.home_mortgage_interest_amount, + "deductible_points_amount": row.deductible_points_amount, + "qualified_mortgage_insurance_premiums_amount": ( + row.qualified_mortgage_insurance_premiums_amount + ), + "investment_interest_amount": row.investment_interest_amount, + "non_mortgage_interest_amount": row.non_mortgage_interest_amount, + "source_cells": row.source_cells, + } + + +def _agi_band_identity(band: PufE19200AgiBand) -> dict[str, object]: + """Return the ordered runtime semantics of one resolved SOI AGI band.""" + + return { + **_component_identity(band), + "label": band.label, + "lower_bound": band.lower_bound, + "upper_bound": band.upper_bound, + "home_mortgage_share": band.home_mortgage_share, + } + + +def puf_e19200_interest_components_asset_identity( + resource: Any | None = None, +) -> dict[str, object]: + """Bind the exact SOI asset bytes and resolved ordered AGI-band semantics.""" + + resolved_resource = ( + files("microcosm.build.us").joinpath(_SOURCE_ASSET) + if resource is None + else resource + ) + all_returns, bands = _load_source_asset(resolved_resource) + return { + "asset": f"microcosm.build.us/{_SOURCE_ASSET}", + "asset_sha256": hashlib.sha256(resolved_resource.read_bytes()).hexdigest(), + "all_returns": _component_identity(all_returns), + "agi_bands": [_agi_band_identity(band) for band in bands], + } + + ( US_PUF_E19200_ALL_RETURNS_COMPONENTS, US_PUF_E19200_AGI_BANDS, @@ -161,6 +218,34 @@ def _load_source_asset() -> tuple[ ) +def puf_e19200_agi_bands_runtime_identity( + bands: Sequence[PufE19200AgiBand] | None = None, +) -> dict[str, object]: + """Bind the exact ordered SOI-band objects consumed by runtime code.""" + + resolved = tuple(US_PUF_E19200_AGI_BANDS if bands is None else bands) + if not resolved: + raise ValueError("PUF E19200 runtime AGI bands must be nonempty.") + if resolved[0].lower_bound is not None or resolved[-1].upper_bound is not None: + raise ValueError("PUF E19200 runtime AGI bands must cover the real line.") + for previous, following in zip(resolved[:-1], resolved[1:], strict=True): + if previous.upper_bound != following.lower_bound: + raise ValueError("PUF E19200 runtime AGI bands must be contiguous.") + payload: dict[str, object] = { + "schema_version": 1, + "agi_bands": [_agi_band_identity(band) for band in resolved], + } + payload["sha256"] = hashlib.sha256( + json.dumps( + payload, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + ).hexdigest() + return payload + + def split_us_puf_e19200_by_agi_band( total_interest_paid: Any, adjusted_gross_income: Any, @@ -216,5 +301,6 @@ def split_us_puf_e19200_by_agi_band( "PufE19200InterestComponents", "US_PUF_E19200_AGI_BANDS", "US_PUF_E19200_ALL_RETURNS_COMPONENTS", + "puf_e19200_interest_components_asset_identity", "split_us_puf_e19200_by_agi_band", ] diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_qrf_chain.py b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_qrf_chain.py index 4ea4ffd5..616c9352 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_qrf_chain.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_qrf_chain.py @@ -456,6 +456,7 @@ def finalize_primary_puf_qrf_chain( checkpoint_dir: str | Path, *, tail_bound_diagnostics: list[dict[str, object]] | None = None, + tail_bound_quantiles: Mapping[str, float] | None = None, ) -> tuple[Frame, str]: """Finalize all raw checkpoints onto ``frame`` and return fit weight kind.""" @@ -480,6 +481,7 @@ def finalize_primary_puf_qrf_chain( person_outputs=_manifest_strings(manifest, "person_outputs"), tax_unit_outputs=_manifest_strings(manifest, "tax_unit_outputs"), tail_bound_diagnostics=tail_bound_diagnostics, + tail_bound_quantiles=tail_bound_quantiles, **_finalization_doctrine_kwargs(manifest), ) initial_state = manifest.get("initial_state") diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py index 2a6f9bbe..c359eaee 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py @@ -77,6 +77,7 @@ "impute_us_puf_tax_detail_support", "puf_tax_detail_clone_mask", "puf_recipient_predictor_universe_receipt", + "puf_tax_detail_tail_bound_quantiles_identity", "puf_tax_unit_donor_from_arrays", "prepare_us_puf_tax_detail_chain_inputs", "resolve_formula_owned_outputs", @@ -308,6 +309,13 @@ class _PredictorSourcePlan: "non_sch_d_capital_gains": 0.999 } + +def puf_tax_detail_tail_bound_quantiles_identity() -> dict[str, float]: + """Return the exact canonical QRF-finalization tail-bound map.""" + + return dict(sorted(_PUF_TAX_DETAIL_TAIL_BOUND_QUANTILES.items())) + + # ASEC directly measures recipient alimony. The PUF QRF therefore sparsifies # only the cloned PUF half for this leaf; pruning the ASEC half would discard # reported source observations. Expense has no ASEC analogue, so its zero ASEC @@ -482,6 +490,14 @@ class _PredictorSourcePlan: "estate_income", ), } +_PERSON_OUTPUT_BOOLEAN_INCIDENCE_DISTRIBUTION_BASES = frozenset( + { + ("qualified_tuition_expenses", "is_full_time_college_student"), + ("sstb_self_employment_income_before_lsr", "business_is_sstb"), + ("sstb_unadjusted_basis_qualified_property", "business_is_sstb"), + ("sstb_w2_wages_from_qualified_business", "business_is_sstb"), + } +) _PUF_EARNINGS_UNIVERSE_PERSON_OUTPUTS = frozenset( { *ACS_PUMS_EARNINGS_SOURCE_COLUMNS, @@ -1565,6 +1581,7 @@ def impute_us_puf_tax_detail_support( fit_records: list[FitWeightRecord] | None = None, raw_predictions_callback: Callable[[pd.DataFrame], None] | None = None, tail_bound_diagnostics: list[dict[str, object]] | None = None, + tail_bound_quantiles: Mapping[str, float] | None = None, predictor_universe_receipts: list[dict[str, object]] | None = None, require_complete_recipient_predictors: bool = False, absent_cells: str = PUF_ABSENT_CELLS_LEGACY_ZERO_FILL, @@ -1692,6 +1709,7 @@ def impute_us_puf_tax_detail_support( person_outputs=person_outputs, tax_unit_outputs=tax_unit_outputs, tail_bound_diagnostics=tail_bound_diagnostics, + tail_bound_quantiles=tail_bound_quantiles, absent_cells=absent_cells, ) @@ -1983,11 +2001,14 @@ def finalize_us_puf_tax_detail_predictions( person_puf_mask=person_puf_mask, ) for column in person_outputs: - _ensure_float_output_column( - tables["person"], - column, - preserve_nulls=preserve_nulls, - ) + if column in _PUF_TAX_DETAIL_BOOLEAN_PERSON_OUTPUTS and preserve_nulls: + _ensure_boolean_output_column(tables["person"], column) + else: + _ensure_float_output_column( + tables["person"], + column, + preserve_nulls=preserve_nulls, + ) allocation_mask = person_puf_mask if column in _PUF_EARNINGS_UNIVERSE_PERSON_OUTPUTS: if earnings_eligible_mask is None: # pragma: no cover - loop invariant @@ -2007,6 +2028,7 @@ def finalize_us_puf_tax_detail_predictions( mask=allocation_mask, column=column, totals=totals, + preserve_boolean_dtype=preserve_nulls, fallback_basis_columns=_PERSON_OUTPUT_DISTRIBUTION_BASIS.get( column, () ), @@ -2018,6 +2040,7 @@ def finalize_us_puf_tax_detail_predictions( column=column, totals=totals, nonnegative=column in _PUF_TAX_DETAIL_NONNEGATIVE_OUTPUTS, + allow_legacy_numeric_boolean_basis=not preserve_nulls, fallback_basis_columns=_PERSON_OUTPUT_DISTRIBUTION_BASIS.get( column, () ), @@ -3237,6 +3260,39 @@ def _ensure_float_output_column( ) +def _ensure_boolean_output_column(table: pd.DataFrame, column: str) -> None: + """Materialize one null-preserving logical boolean without numeric coercion.""" + + if column not in table.columns: + table[column] = pd.Series( + pd.array([pd.NA] * len(table), dtype="boolean"), + index=table.index, + ) + return + + values = table[column] + observed = values.dropna() + invalid = observed.map(lambda value: not isinstance(value, (bool, np.bool_))) + if invalid.any(): + offending_types = sorted( + { + f"{type(value).__module__}.{type(value).__qualname__}" + for value in observed.loc[invalid] + } + ) + raise TypeError( + f"PUF boolean output {column!r} must contain only physical boolean " + "values before null-preserving materialization; got " + f"dtype {values.dtype!s} with offending value types " + f"{offending_types}." + ) + table[column] = pd.Series( + pd.array(values, dtype="boolean"), + index=table.index, + name=column, + ) + + def _snap_to_observed_values( values: Sequence[Any], observed: Sequence[Any], @@ -3303,12 +3359,118 @@ def _puf_earnings_allocation_mask( return person_puf_mask & age.ge(ACS_PUMS_EARNINGS_MINIMUM_AGE) +def _nonnegative_allocation_basis_values( + values: pd.Series, + *, + output_column: str, + basis_column: str, + allow_legacy_numeric_boolean: bool = False, +) -> np.ndarray: + """Return temporary numeric weights without changing a basis column's dtype. + + PUF person outputs can be distributed using canonical boolean incidence, + including ``is_full_time_college_student`` for tuition and + ``business_is_sstb`` for QBI amounts. Nullable booleans cannot accept a + floating fill value, so map that declared incidence semantics to 0/1 only + in the transient allocation vector. The stored column remains a physical + boolean and is still checked as such by the late-output guard. + """ + + observed = values.dropna() + observed_boolean = observed.map(lambda value: isinstance(value, (bool, np.bool_))) + physical_boolean = bool( + pd.api.types.is_bool_dtype(values.dtype) + or (len(observed) and observed_boolean.all()) + ) + contains_physical_boolean = bool( + pd.api.types.is_bool_dtype(values.dtype) or observed_boolean.any() + ) + expects_boolean_incidence = ( + output_column, + basis_column, + ) in _PERSON_OUTPUT_BOOLEAN_INCIDENCE_DISTRIBUTION_BASES + if expects_boolean_incidence and physical_boolean: + numeric = pd.Series( + pd.array(values, dtype="boolean").astype("Float64"), + index=values.index, + ) + else: + if expects_boolean_incidence and not allow_legacy_numeric_boolean: + offending_types = sorted( + { + f"{type(value).__module__}.{type(value).__qualname__}" + for value in observed + } + ) + raise TypeError( + f"PUF output {output_column!r} allocation basis " + f"{basis_column!r} declares boolean_incidence and must " + "contain only physical booleans; got " + f"dtype {values.dtype!s} with observed value types " + f"{offending_types}." + ) + if not expects_boolean_incidence and contains_physical_boolean: + raise TypeError( + f"PUF output {output_column!r} allocation basis " + f"{basis_column!r} declares monetary_sign_separated and " + "cannot contain physical booleans." + ) + invalid = observed.map( + lambda value: ( + isinstance(value, (bool, np.bool_)) + or not isinstance( + value, + (int, float, np.integer, np.floating), + ) + ) + ) + if invalid.any(): + offending_types = sorted( + { + f"{type(value).__module__}.{type(value).__qualname__}" + for value in observed.loc[invalid] + } + ) + raise TypeError( + f"PUF output {output_column!r} allocation basis " + f"{basis_column!r} declares monetary_sign_separated and must " + f"contain only real numeric values; got dtype {values.dtype!s} with " + f"offending value types {offending_types}." + ) + nonfinite = observed.map(lambda value: not np.isfinite(float(value))) + if nonfinite.any(): + raise ValueError( + f"PUF output {output_column!r} allocation basis " + f"{basis_column!r} contains {int(nonfinite.sum())} " + "nonfinite observed value(s)." + ) + numeric = pd.Series( + pd.array(values, dtype="Float64"), + index=values.index, + ) + if expects_boolean_incidence: + outside_boolean_support = ~numeric.dropna().isin([0.0, 1.0]) + if outside_boolean_support.any(): + raise ValueError( + f"PUF output {output_column!r} legacy allocation basis " + f"{basis_column!r} declares boolean_incidence but contains " + f"{int(outside_boolean_support.sum())} numeric value(s) " + "outside exact {0, 1} support." + ) + return np.clip( + numeric.to_numpy(dtype=np.float64, na_value=0.0), + 0.0, + None, + ) + + def _write_person_tax_unit_boolean_counts( person: pd.DataFrame, *, mask: pd.Series, column: str, totals: pd.Series, + preserve_boolean_dtype: bool = False, fallback_basis_columns: tuple[str, ...] = (), ) -> None: """Place a predicted number of true people within each tax unit. @@ -3328,11 +3490,10 @@ def _write_person_tax_unit_boolean_counts( for basis_column in fallback_basis_columns: if basis_column not in person.columns: continue - score += ( - pd.to_numeric(person.loc[mask, basis_column], errors="coerce") - .fillna(0.0) - .clip(lower=0.0) - .to_numpy(dtype=np.float64) + score += _nonnegative_allocation_basis_values( + person.loc[mask, basis_column], + output_column=column, + basis_column=basis_column, ) placement = pd.DataFrame( @@ -3365,7 +3526,10 @@ def _write_person_tax_unit_boolean_counts( ) placement["selected"] = placement["rank"].to_numpy() < desired selected = placement["selected"].reindex(row_ids.index).fillna(False) - person.loc[mask, column] = selected.to_numpy(dtype=np.float64) + selected_values = selected.to_numpy(dtype=bool) + if not preserve_boolean_dtype: + selected_values = selected_values.astype(np.float64) + person.loc[mask, column] = selected_values def _write_person_tax_unit_totals( @@ -3375,6 +3539,7 @@ def _write_person_tax_unit_totals( column: str, totals: pd.Series, nonnegative: bool, + allow_legacy_numeric_boolean_basis: bool = False, fallback_basis_columns: tuple[str, ...] = (), ) -> None: row_ids = person.loc[mask, "person_tax_unit_id"] @@ -3397,11 +3562,11 @@ def _write_person_tax_unit_totals( for basis_column in fallback_basis_columns: if basis_column not in person.columns: continue - fallback += ( - pd.to_numeric(person.loc[mask, basis_column], errors="coerce") - .fillna(0.0) - .clip(lower=0.0) - .to_numpy(dtype=np.float64) + fallback += _nonnegative_allocation_basis_values( + person.loc[mask, basis_column], + output_column=column, + basis_column=basis_column, + allow_legacy_numeric_boolean=allow_legacy_numeric_boolean_basis, ) fallback_sum = ( pd.Series(fallback, index=row_ids.index) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py b/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py index 3d29047d..6b4ffc1f 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py @@ -35,18 +35,26 @@ from __future__ import annotations import hashlib +import importlib import json import math +import os import pickle +import struct +import sys from collections import Counter -from collections.abc import Mapping, Sequence -from dataclasses import dataclass, field +from collections.abc import Callable, Mapping, Sequence +from dataclasses import asdict, dataclass, field +from importlib.resources import files from pathlib import Path from types import MappingProxyType import numpy as np import pandas as pd +import microcosm.build.us_runtime.acs_income_universe as acs_income_universe_runtime +import microcosm.build.us_runtime.acs_transfer as acs_transfer_runtime +import microcosm.build.us_runtime.multispine_pool as multispine_pool_runtime from microcosm.build.frame_sampling import ( EXACT_COUNT_RULE as _EXACT_COUNT_RULE, ) @@ -71,20 +79,42 @@ _sealed_stacked_gate_result, ) from microcosm.build.serialization_dtypes import canonicalize_table_string_dtypes +from microcosm.build.source_manifest import load_source_manifest from microcosm.build.us_runtime.acs_income_universe import ( + ACS_PUMS_EARNINGS_SOURCE_COLUMNS, + AcsPumsEarningsUniverseApplication, + acs_pums_earnings_universe_contract_identity, apply_acs_pums_earnings_universe_zeros, + resolve_acs_pums_earnings_universe, ) from microcosm.build.us_runtime.acs_transfer import ( + ASEC_PUF_DONOR_SPINE, DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, AcsTransferResult, AcsTransferTargetBank, TargetFamilies, transfer_acs_inputs, ) +from microcosm.build.us_runtime.late_producer_dag import ( + ProducerContract, + ProducerInput, + ProducerInputColumn, + ProducerOutput, + run_producer_when_ready, +) from microcosm.build.us_runtime.multispine_pool import ( + POOL_DEFERRED_TRANSFER_INPUTS, + POOL_DEFERRED_TRANSFER_STATUS, + POOL_HOUSING_ASSISTANCE_MAX_TRAIN_SAMPLES, + POOL_HOUSING_ASSISTANCE_N_ESTIMATORS, POOL_OPERATOR_CONTRACTS, + POOL_POST_CLONE_SOURCE_OPERATOR_ORDER, + POOL_POST_CLONE_SOURCE_PHASE, POOL_PRE_CLONE_SOURCE_OPERATOR_ORDER, + POOL_RANDOM_SEED, + POOL_SOURCE_ALLOW_EXISTING_WITHOUT_SOURCE, POOL_SPINE_AGREEMENT_REGISTRY, + POOL_TIME_PERIOD, pool_post_puf_puf_producer_target_families, pool_post_puf_source_producer_target_families, pool_post_puf_transfer_target_families, @@ -92,8 +122,12 @@ pool_transfer_target_families, ) from microcosm.build.us_runtime.operator_boundary import ( + FORMULA_OWNED_SOURCE_COLUMNS, PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES, ) +from microcosm.build.us_runtime.puf_aggregate_records import ( + PufAggregateDisaggregationSpec, +) from microcosm.build.us_runtime.puf_capital_gains_tail import ( PUF_CAPITAL_GAINS_TAIL_APPLIED_COLUMN, PUF_CAPITAL_GAINS_TAIL_DONOR_AGI_BAND_COLUMN, @@ -104,11 +138,21 @@ PUF_CAPITAL_GAINS_TAIL_SUPPORT_CHANNEL, PUF_CAPITAL_GAINS_TAIL_TAX_UNIT_COLUMNS, PUF_CAPITAL_GAINS_TAIL_TRANSFER_WEIGHT_COLUMN, + puf_capital_gains_tail_execution_inputs_identity, + puf_capital_gains_tail_spec_identity, + puf_capital_gains_tail_support_contract_identity, + puf_capital_gains_tail_terminal_support_receipt, + resolve_puf_capital_gains_tail_execution_inputs, transfer_puf_capital_gains_tail, validate_puf_capital_gains_tail_manifest, + validate_puf_capital_gains_tail_terminal_support_receipt, ) +from microcosm.build.us_runtime.puf_interest_components import PufE19200AgiBand from microcosm.build.us_runtime.puf_qrf_chain import ( + PRIMARY_QRF_CHECKPOINT_SCHEMA_VERSION, PRIMARY_QRF_MANIFEST_FILENAME, + PRIMARY_QRF_TARGET_ORDER, + PRIMARY_QRF_TARGET_ORDER_SHA256, finalize_primary_puf_qrf_chain, initialize_primary_puf_qrf_chain, primary_puf_qrf_recipient_predictor_universe_receipt, @@ -118,11 +162,14 @@ PUF_ABSENT_CELLS_PRESERVE_NULLS, PUF_CLONE_ATTACHMENT_MANIFEST_KEY, PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS, + PUF_TAX_DETAIL_DEFAULT_PREDICTORS, PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS, + PUF_TAX_DETAIL_SUPPORT_CHANNEL, US_PUF_SUPPORT_FIT_NAME, bind_puf_clone_attachment_tail_descendant, clone_us_frame_for_puf_support, impute_us_puf_tax_detail_support, + puf_tax_detail_tail_bound_quantiles_identity, validate_puf_clone_attachment, ) from microcosm.build.us_runtime.spine_assembly import assemble_spines @@ -136,6 +183,30 @@ support_source_id_column, validate_assembly_provenance, ) +from microcosm.build.us_runtime.us_late_overlap_ownership import ( + us_late_overlap_ownership_receipt, + validate_us_late_overlap_ownership_receipt, +) +from microcosm.build.us_runtime.us_late_producer_registry import ( + CANONICAL_US_LATE_PRODUCER_REGISTRY, + CANONICAL_US_LATE_PRODUCER_SCHEDULE, + CANONICAL_US_LATE_TRANSFER_GROUPS, + US_LATE_ACS_EARNINGS_UNIVERSE_CONFIG_INPUT, + US_LATE_ACS_EARNINGS_UNIVERSE_RECEIPT_INPUT, + US_LATE_ACS_EARNINGS_UNIVERSE_STAGE, + US_LATE_PRIMARY_EXECUTION_CONFIG_INPUT, + US_LATE_PRIMARY_PUF_STAGE, + US_LATE_PRODUCER_RECEIPT_SCHEMA_VERSION, + US_LATE_PRODUCER_TRANSITION_AUTHORITY_ID, + US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY, + US_LATE_PRODUCER_TRANSITION_AUTHORITY_VERSION, + US_LATE_SOURCE_EXECUTION_CONFIG_INPUT, + US_LATE_SOURCE_FINALIZER_EXECUTION_CONFIG_INPUT, + US_LATE_SOURCE_FINALIZER_STAGE, + US_LATE_TRANSFER_MODEL_CONFIG_INPUT, + US_LATE_TRANSFER_TARGET_BANK_INPUT, + us_late_producer_schedule_receipt, +) from microcosm.frame import US_SCHEMA, Frame __all__ = [ @@ -154,12 +225,15 @@ "STACKED_PILOT_ACS_SAMPLE_FRACTION", "STACKED_PILOT_ACS_SAMPLE_SEED", "STACKED_SPINE_MANIFEST_KEY", + "US_PUF_S_CORP_UNIVERSE_ZERO_RULE_ID", + "US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY", "AbsenceProof", "GapFillAbsenceRule", "GapFillDirection", "GapFillResult", "OriginBatterySpec", "StackedPufPassResult", + "StackedLateProducerResult", "StackedPostPufTransferResult", "StackedSpineResult", "assemble_stacked_spine", @@ -167,13 +241,21 @@ "by_origin_battery", "gap_fill_stacked_spine", "run_stacked_puf_pass", + "run_stacked_late_producer_dag", "prepare_stacked_tail_derivation", "sample_acs_households", "stacked_completeness_gate", "stacked_gap_fill_plan", "stacked_gap_fill_producer_schedule_receipt", + "stacked_late_primary_checkpoint_input_binding", + "stacked_late_primary_resource_receipts", + "stacked_late_producer_resource_semantics_receipt", "stacked_spine_authority_receipt", "transfer_stacked_post_puf_inputs", + "transfer_stacked_post_puf_group", + "us_puf_s_corp_universe_zero_rule_identity", + "validate_stacked_late_producer_receipt", + "validate_stacked_late_producer_transition_authority", "validate_stacked_post_puf_transfer_receipt", "validate_stacked_spine_frame", ] @@ -190,6 +272,7 @@ } STACKED_SPINE_MANIFEST_KEY = "us_stacked_spine_manifest" +US_PUF_S_CORP_UNIVERSE_ZERO_RULE_ID = "puf_tax_detail_s_corp_income_universe_zero_v1" _LEGACY_STACKED_SPINE_MANIFEST_VERSION = 1 _STACKED_SPINE_MANIFEST_VERSION = 4 _SUPPORTED_STACKED_SPINE_MANIFEST_VERSIONS = { @@ -1577,6 +1660,17 @@ def thaw(item: object) -> object: return {str(key): thaw(nested) for key, nested in item.items()} if isinstance(item, (list, tuple)): return [thaw(nested) for nested in item] + if isinstance(item, (set, frozenset)): + values = [thaw(nested) for nested in item] + return sorted( + values, + key=lambda nested: json.dumps( + nested, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ), + ) if isinstance(item, np.generic): return item.item() return item @@ -1594,9 +1688,11 @@ def thaw(item: object) -> object: _GAP_FILL_ASEC_HOUSING_TO_ACS = "asec_housing_to_acs" _GAP_FILL_HOUSING_FAMILY = "housing" _STACKED_AUTHORITY_ID = "us_stacked_spine_authority" -# v6 binds the ASEC-consistent ACS earnings-universe application and the -# authenticated whole-pool QBI mutation semantics into the outer identity. -_STACKED_AUTHORITY_VERSION = 6 +# v10 binds the primary-PUF whole-pool output-universe declaration. v9 bound +# the content-hashed execution/transition-authority schema in addition to the +# import-validated producer/input DAG. Version 8 named the graph but did not +# authenticate its live input/output transition. +_STACKED_AUTHORITY_VERSION = 10 _CANONICAL_AUTHORITY_FORM = "CANONICAL" _NONCANONICAL_AUTHORITY_FORM = "NON-CANONICAL" _PRE_CLONE_PREPARATION_STAGE = "prepare_multispine_source_inputs_for_clone" @@ -1815,6 +1911,28 @@ class _BatterySupportProfile: min_effective_support: int +def _freeze_authority_payload(value: object) -> object: + """Recursively freeze one JSON-shaped authority component.""" + + if isinstance(value, Mapping): + return MappingProxyType( + { + str(key): _freeze_authority_payload(nested) + for key, nested in value.items() + } + ) + if isinstance(value, (list, tuple)): + return tuple(_freeze_authority_payload(nested) for nested in value) + if value is None or isinstance(value, (str, int, float, bool)): + return value + if isinstance(value, np.generic): + return value.item() + raise TypeError( + "Stacked authority components must contain only canonical JSON values; " + f"got {type(value).__name__}." + ) + + @dataclass(frozen=True) class _StackedAuthority: """One digest-carrying, deeply immutable stacked-spine authority bundle.""" @@ -1829,6 +1947,8 @@ class _StackedAuthority: metric_registry: Mapping[tuple[str, str, str, int], str] joint_metric_registry: Mapping[tuple[str, str, tuple[str, ...], int], str] support_profile: _BatterySupportProfile + puf_capital_gains_tail_support_contract: Mapping[str, object] + late_producer_schedule: Mapping[str, object] declared_component_sha256: Mapping[str, str] declared_sha256: str declared_form: str @@ -1876,6 +1996,25 @@ def __post_init__(self) -> None: raise TypeError( "Stacked authority support_profile must be a _BatterySupportProfile." ) + if not isinstance(self.puf_capital_gains_tail_support_contract, Mapping): + raise TypeError( + "Stacked authority capital-gains-tail support contract must be " + "a mapping." + ) + object.__setattr__( + self, + "puf_capital_gains_tail_support_contract", + _freeze_authority_payload(self.puf_capital_gains_tail_support_contract), + ) + if not isinstance(self.late_producer_schedule, Mapping): + raise TypeError( + "Stacked authority late producer schedule must be a mapping." + ) + object.__setattr__( + self, + "late_producer_schedule", + _freeze_authority_payload(self.late_producer_schedule), + ) component_digests = dict(self.declared_component_sha256) if set(component_digests) != { "gap_fill_plan", @@ -1884,6 +2023,8 @@ def __post_init__(self) -> None: "metric_registry", "joint_metric_registry", "support_profile", + "puf_capital_gains_tail_support_contract", + "late_producer_schedule", }: raise ValueError( "Stacked authority must carry every component's declared digest." @@ -2177,6 +2318,8 @@ def _authority_component_payloads( metric_registry: Mapping[tuple[str, str, str, int], str], joint_metric_registry: Mapping[tuple[str, str, tuple[str, ...], int], str], support_profile: _BatterySupportProfile, + puf_capital_gains_tail_support_contract: Mapping[str, object], + late_producer_schedule: Mapping[str, object], ) -> dict[str, object]: return { "gap_fill_plan": _plan_payload(gap_fill_plan), @@ -2196,6 +2339,10 @@ def _authority_component_payloads( "metric_registry": _metric_registry_payload(metric_registry), "joint_metric_registry": _joint_metric_registry_payload(joint_metric_registry), "support_profile": _support_profile_payload(support_profile), + "puf_capital_gains_tail_support_contract": _json_ready( + puf_capital_gains_tail_support_contract + ), + "late_producer_schedule": _json_ready(late_producer_schedule), } @@ -2222,6 +2369,10 @@ def _authority_live_digests( metric_registry=authority.metric_registry, joint_metric_registry=authority.joint_metric_registry, support_profile=authority.support_profile, + puf_capital_gains_tail_support_contract=( + authority.puf_capital_gains_tail_support_contract + ), + late_producer_schedule=authority.late_producer_schedule, ) component_digests = { name: _canonical_sha256(payload) for name, payload in payloads.items() @@ -2248,6 +2399,8 @@ def _make_stacked_authority( metric_registry: Mapping[tuple[str, str, str, int], str], support_profile: _BatterySupportProfile, declared_form: str, + puf_capital_gains_tail_support_contract: Mapping[str, object] | None = None, + late_producer_schedule: Mapping[str, object] | None = None, joint_metric_registry: Mapping[tuple[str, str, tuple[str, ...], int], str] | None = None, declared_component_sha256: Mapping[str, str] | None = None, @@ -2266,6 +2419,20 @@ def _make_stacked_authority( frozen_joint_registry = _freeze_joint_metric_registry( {} if joint_metric_registry is None else joint_metric_registry ) + frozen_tail_support_contract = _freeze_authority_payload( + puf_capital_gains_tail_support_contract_identity() + if puf_capital_gains_tail_support_contract is None + else puf_capital_gains_tail_support_contract + ) + if not isinstance(frozen_tail_support_contract, Mapping): + raise TypeError("Capital-gains-tail support contract must be a mapping.") + frozen_late_producer_schedule = _freeze_authority_payload( + us_late_producer_schedule_receipt() + if late_producer_schedule is None + else late_producer_schedule + ) + if not isinstance(frozen_late_producer_schedule, Mapping): + raise TypeError("Late producer schedule must be a mapping.") component_payloads = _authority_component_payloads( gap_fill_plan=frozen_plan, post_puf_transfer_surface=frozen_post_puf_surface, @@ -2275,6 +2442,8 @@ def _make_stacked_authority( metric_registry=frozen_registry, joint_metric_registry=frozen_joint_registry, support_profile=support_profile, + puf_capital_gains_tail_support_contract=frozen_tail_support_contract, + late_producer_schedule=frozen_late_producer_schedule, ) live_components = { name: _canonical_sha256(payload) for name, payload in component_payloads.items() @@ -2297,6 +2466,8 @@ def _make_stacked_authority( metric_registry=frozen_registry, joint_metric_registry=frozen_joint_registry, support_profile=support_profile, + puf_capital_gains_tail_support_contract=frozen_tail_support_contract, + late_producer_schedule=frozen_late_producer_schedule, declared_component_sha256=( live_components if declared_component_sha256 is None @@ -2940,6 +3111,7 @@ def _production_stacked_authority( CANONICAL_ORIGIN_BATTERY_SUPPORT_PROFILE ), ) -> _StackedAuthority: + live_late_producer_schedule = us_late_producer_schedule_receipt() identity = ( _STACKED_GAP_FILL_PLAN is _canonical_plan and _STACKED_GAP_FILL_SURFACE is _canonical_gap_surface @@ -2952,6 +3124,8 @@ def _production_stacked_authority( and _BATTERY_METRIC_REGISTRY is _canonical_registry and _BATTERY_JOINT_METRIC_REGISTRY is _canonical_joint_registry and _BATTERY_SUPPORT_PROFILE is _canonical_profile + and _json_ready(live_late_producer_schedule) + == _json_ready(_canonical_authority.late_producer_schedule) ) if identity: return _canonical_authority @@ -2966,6 +3140,7 @@ def _production_stacked_authority( metric_registry=_BATTERY_METRIC_REGISTRY, joint_metric_registry=_BATTERY_JOINT_METRIC_REGISTRY, support_profile=_BATTERY_SUPPORT_PROFILE, + late_producer_schedule=live_late_producer_schedule, declared_form=_CANONICAL_AUTHORITY_FORM, declared_component_sha256=_canonical_authority.declared_component_sha256, declared_sha256=_canonical_authority.declared_sha256, @@ -3256,6 +3431,26 @@ def _authority_receipt( "declared_sha256": authority.declared_component_sha256["support_profile"], "digest_matches_declared": component_integrity["support_profile"], }, + "puf_capital_gains_tail_support_contract": { + "identity": _json_ready(authority.puf_capital_gains_tail_support_contract), + "sha256": live_components["puf_capital_gains_tail_support_contract"], + "declared_sha256": authority.declared_component_sha256[ + "puf_capital_gains_tail_support_contract" + ], + "digest_matches_declared": component_integrity[ + "puf_capital_gains_tail_support_contract" + ], + }, + "late_producer_schedule": { + "identity": _json_ready(authority.late_producer_schedule), + "sha256": live_components["late_producer_schedule"], + "declared_sha256": authority.declared_component_sha256[ + "late_producer_schedule" + ], + "schedule_sha256": authority.late_producer_schedule.get("schedule_sha256"), + "producer_count": authority.late_producer_schedule.get("producer_count"), + "digest_matches_declared": component_integrity["late_producer_schedule"], + }, } return { "authority_id": authority.authority_id, @@ -3403,6 +3598,11 @@ def _authority_validation_failures( ("metric_registry", "metric registry"), ("joint_metric_registry", "joint metric registry"), ("support_profile", "support profile"), + ( + "puf_capital_gains_tail_support_contract", + "PUF capital-gains-tail support contract", + ), + ("late_producer_schedule", "late producer schedule"), ): component = receipt["components"][name] if not component["digest_matches_declared"]: @@ -3501,7 +3701,7 @@ def validate_stacked_post_puf_transfer_receipt( *, boundary: str, ) -> None: - """Reject a late-transfer receipt unless it carries canonical authority.""" + """Reject a late-transfer receipt unless its full DAG proof is canonical.""" if not isinstance(receipt, Mapping): raise ValueError(f"{boundary}: stacked post-PUF transfer receipt is absent.") @@ -3512,151 +3712,3492 @@ def validate_stacked_post_puf_transfer_receipt( "production manifest emission is forbidden." ) _validate_production_authority_receipt(authority, boundary=boundary) + schedule = receipt.get("producer_schedule") + expected_schedule = _json_ready(us_late_producer_schedule_receipt()) + if not isinstance(schedule, Mapping) or _json_ready(schedule) != expected_schedule: + raise ValueError( + f"{boundary}: stacked post-PUF transfer receipt has no canonical " + "late-producer schedule; production manifest emission is forbidden." + ) + expected_execution_order = [ + producer + for producer in CANONICAL_US_LATE_PRODUCER_SCHEDULE.order + if producer != US_LATE_PRIMARY_PUF_STAGE + ] + if receipt.get("producer_execution_order") != expected_execution_order: + raise ValueError( + f"{boundary}: stacked post-PUF transfer execution order does not " + "match the derived late-producer schedule; production manifest " + "emission is forbidden." + ) + expected_groups = {group.name: group for group in CANONICAL_US_LATE_TRANSFER_GROUPS} + groups = receipt.get("groups") + if not isinstance(groups, Mapping) or set(groups) != set(expected_groups): + raise ValueError( + f"{boundary}: stacked post-PUF transfer group surface is not the " + "canonical 19-group partition; production manifest emission is " + "forbidden." + ) + for name, group in expected_groups.items(): + group_receipt = groups[name] + if ( + not isinstance(group_receipt, Mapping) + or group_receipt.get("producer") != name + or tuple(group_receipt.get("ordered_targets", ())) != group.targets + ): + raise ValueError( + f"{boundary}: stacked post-PUF transfer group {name!r} is " + "misbound; production manifest emission is forbidden." + ) + expected_target_labels = { + f"{entity}/{family}/{target}" + for entity, families in CANONICAL_STACKED_POST_PUF_TRANSFER_SURFACE.items() + for family, targets in families.items() + for target in targets + } + targets = receipt.get("targets") + if not isinstance(targets, Mapping) or set(targets) != expected_target_labels: + raise ValueError( + f"{boundary}: stacked post-PUF transfer target surface is not the " + "canonical 70-target surface; production manifest emission is " + "forbidden." + ) + if any( + not isinstance(target_receipt, Mapping) + or target_receipt.get("residual_null_rows") != 0 + for target_receipt in targets.values() + ): + raise ValueError( + f"{boundary}: stacked post-PUF transfer target receipts do not " + "prove zero residual nulls; production manifest emission is forbidden." + ) + completion = receipt.get("completion") + if completion != { + "status": "complete", + "group_count": 19, + "target_count": 70, + "residual_null_rows": 0, + }: + raise ValueError( + f"{boundary}: stacked post-PUF transfer completion receipt is not " + "canonical; production manifest emission is forbidden." + ) -def _validate_test_authority(authority: _StackedAuthority, *, boundary: str) -> None: - """Keep the explicit fixture seam visibly and terminally non-production.""" +_LATE_TABLE_DIGEST_CODEC = "canonical_scalar_v1" +_LATE_PRIMARY_QRF_INPUT_BINDING_ARTIFACT_KIND = ( + "populace_us_stacked_late_primary_qrf_input_binding" +) +_LATE_PRIMARY_QRF_INPUT_BINDING_FILENAME = "late-producer-input-binding.json" +_LATE_RESOURCE_SEMANTICS_ARTIFACT_KIND = ( + "populace_us_stacked_late_producer_resource_semantics" +) +_LATE_TABLE_DIGEST_CHUNK_ROWS = 65_536 - receipt = _authority_receipt(authority) - if ( - authority.declared_form != _NONCANONICAL_AUTHORITY_FORM - or receipt["authority_form"] != _NONCANONICAL_AUTHORITY_FORM - or receipt["canonical_identity"] is not False + +def _late_digest_part( + digest, + *, + domain: str, + payload: bytes | bytearray | memoryview | np.ndarray, +) -> None: + """Append one length-framed, domain-separated byte field to a digest.""" + + domain_bytes = domain.encode("utf-8") + payload_view = memoryview(payload) + if payload_view.format != "B" or payload_view.ndim != 1: + payload_view = payload_view.cast("B") + digest.update(struct.pack(" np.ndarray: + """Return a contiguous, explicitly little-endian numeric byte source.""" + + array = np.asarray(values) + array = array.astype(array.dtype.newbyteorder("<"), copy=False) + return np.ascontiguousarray(array) + + +def _late_scalar_bytes(value: object) -> bytes: + """Encode one supported object scalar without lossy intermediary hashes.""" + + missing = pd.isna(value) + if isinstance(missing, (bool, np.bool_)) and bool(missing): + return b"null" + if isinstance(value, (bool, np.bool_)): + return b"bool\x01" if bool(value) else b"bool\x00" + if isinstance(value, (int, np.integer)): + return b"integer\x00" + str(int(value)).encode("ascii") + if isinstance(value, (float, np.floating)): + if isinstance(value, np.floating) and value.dtype.itemsize > 8: + raise TypeError( + "US late-producer content digest does not support object " + f"floating scalar {value.dtype!s}." + ) + return b"float64\x00" + struct.pack(" 16: + raise TypeError( + "US late-producer content digest does not support object " + f"complex scalar {value.dtype!s}." + ) + numeric = complex(value) + return b"complex128\x00" + struct.pack(" None: + """Stream framed string or object scalars in bounded-memory chunks.""" + + for chunk_index, start in enumerate( + range(0, len(values), _LATE_TABLE_DIGEST_CHUNK_ROWS) ): - raise ValueError(f"{boundary} requires a NON-CANONICAL test authority.") + stop = min(start + _LATE_TABLE_DIGEST_CHUNK_ROWS, len(values)) + lengths = np.zeros(stop - start, dtype=" None: - """Validate gate-specific receipts against the captured canonical doctrine.""" + """Hash one ordered logical Series with explicit dtype and null domains.""" + + dtype = series.dtype + missing = series.isna().to_numpy(dtype=bool) + _late_digest_part( + digest, + domain=f"{domain}/dtype", + payload=str(dtype).encode("utf-8"), + ) + _late_digest_part( + digest, + domain=f"{domain}/row_count", + payload=struct.pack(" str: + """Hash ordered table scalars directly with typed, null-aware framing.""" + + values = ( + canonicalize_table_string_dtypes( + table, + boundary="late-producer content digest", + table_name="declared_surface", ) - _validate_production_authority_receipt(authority, boundary=boundary) - authority_sha256 = authority["sha256"] - plan_sha256 = authority["components"]["gap_fill_plan"]["sha256"] - post_puf_surface_sha256 = authority["components"]["post_puf_transfer_surface"][ - "sha256" - ] - surface_sha256 = authority["components"]["declared_surface"]["sha256"] - expected_keys = _surface_target_keys(_canonical_surface) + if normalize_strings + else table + ) + if isinstance(values.index, pd.MultiIndex): + index_levels = [ + pd.Series(values.index.get_level_values(level), copy=False) + for level in range(values.index.nlevels) + ] + else: + index_levels = [pd.Series(values.index, copy=False)] + header = { + "codec": _LATE_TABLE_DIGEST_CODEC, + "columns": [str(column) for column in values.columns], + "dtypes": [ + str(values.iloc[:, index].dtype) for index in range(values.shape[1]) + ], + "index_type": type(values.index).__name__, + "index_dtype": str(values.index.dtype), + "index_level_dtypes": [str(level.dtype) for level in index_levels], + "index_names": [ + None if name is None else str(name) for name in values.index.names + ], + } + digest = hashlib.sha256() + _late_digest_part( + digest, + domain="late_table_digest_codec", + payload=_LATE_TABLE_DIGEST_CODEC.encode("ascii"), + ) + _late_digest_part( + digest, + domain="late_table_header_json", + payload=json.dumps( + header, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8"), + ) + for level_index, level in enumerate(index_levels): + _late_digest_series_values( + digest, + level, + domain=f"index_level/{level_index}", + ) + for column_index in range(values.shape[1]): + _late_digest_series_values( + digest, + values.iloc[:, column_index], + domain=f"column/{column_index}", + ) + return digest.hexdigest() - def reject(reason: str) -> None: + +def _late_virtual_resource_kind(column: str) -> str: + """Return the exact semantic kind for one declared virtual input.""" + + kinds = { + "@puf_donor_tax_units": "puf_donor_tax_units", + "@primary_qrf_checkpoint": "primary_qrf_checkpoint", + US_LATE_ACS_EARNINGS_UNIVERSE_CONFIG_INPUT: ( + "acs_pums_earnings_universe_execution_config" + ), + US_LATE_PRIMARY_EXECUTION_CONFIG_INPUT: "primary_puf_execution_config", + US_LATE_SOURCE_EXECUTION_CONFIG_INPUT: ("post_clone_source_execution_config"), + US_LATE_SOURCE_FINALIZER_EXECUTION_CONFIG_INPUT: ( + "source_finalizer_execution_config" + ), + US_LATE_TRANSFER_MODEL_CONFIG_INPUT: "late_transfer_model_config", + US_LATE_TRANSFER_TARGET_BANK_INPUT: "late_transfer_target_bank", + } + if column.startswith("@source_receipt:"): + return "source_operator_receipt" + try: + return kinds[column] + except KeyError as exc: raise ValueError( - f"{boundary}: {reason}; production manifest emission is forbidden." - ) + f"Unknown US late-producer virtual resource input {column!r}." + ) from exc - def nonnegative_int( - receipt: Mapping[str, object], - field_name: str, - *, - label: str, - ) -> int: - value = receipt.get(field_name) + +def _late_resource_binding_schema_version(column: str) -> int: + """Return the independently versioned payload schema for one resource.""" + + kind = _late_virtual_resource_kind(column) + return { + "acs_pums_earnings_universe_execution_config": 2, + "primary_puf_execution_config": 4, + "post_clone_source_execution_config": 3, + "source_finalizer_execution_config": 2, + "late_transfer_model_config": 3, + }.get(kind, 1) + + +def _late_contract_available_input_keys( + contract: ProducerContract, +) -> set[str]: + """Return the exact external-receipt keys required by one contract.""" + + return { + f"{column.entity}.{column.column}" + for requirement in contract.inputs + for alternative in requirement.alternatives + for column in alternative + if column.column.startswith("@") + and column.column != "@resolved_weight" + and column.entity != "frame" + } + + +def _validate_late_resource_binding( + binding: Mapping[str, object], + *, + producer: str, + entity: str, + column: str, + boundary: str, +) -> None: + """Reject hash-consistent resource claims with incomplete semantics.""" + + kind = _late_virtual_resource_kind(column) + + def require_keys(expected: set[str]) -> None: + if set(binding) != expected: + raise ValueError( + f"{boundary}: late resource {entity}.{column} {kind!r} binding " + f"schema drifted; missing={sorted(expected - set(binding))}, " + f"extra={sorted(set(binding) - expected)}." + ) + + def require_nonnegative_integer(value: object, *, label: str) -> int: if isinstance(value, bool) or not isinstance(value, int) or value < 0: - reject(f"{label} {field_name} must be a non-negative integer") + raise ValueError( + f"{boundary}: late resource {entity}.{column} has invalid " + f"{label}={value!r}." + ) return value - def validate_structural_absence_receipt( - raw_receipt: object, - *, - label: str, - battery: bool, - ) -> tuple[int, dict[str, int]]: - if not isinstance(raw_receipt, Mapping): - reject(f"{label} must carry canonical recipient-absence authority") - expected_fields = { - "rule_id", - "selection", - "reason", - "status", - "rows", - "by_origin_role", - "unexpected_null_rows", - "structural_rows_filled", - } - if battery: - expected_fields.update( - {"comparison_clone_index", "rows_excluded_from_scope"} + def require_positive_integer(value: object, *, label: str) -> int: + result = require_nonnegative_integer(value, label=label) + if result == 0: + raise ValueError( + f"{boundary}: late resource {entity}.{column} requires positive " + f"{label}." ) - if set(raw_receipt) != expected_fields: - reject(f"{label} structural-absence receipt schema mismatch") + return result + + common = {"resource_kind", "schema_version"} + if kind == "puf_donor_tax_units": + require_keys({*common, "table_content_sha256", "ordered_columns", "dtypes"}) + _validate_sha256( + binding.get("table_content_sha256"), + boundary=f"{boundary} PUF donor content", + ) + columns = binding.get("ordered_columns") + dtypes = binding.get("dtypes") if ( - raw_receipt.get("rule_id") != _ACS_GQ_RENT_ABSENCE_RULE_ID - or raw_receipt.get("selection") != _ACS_GQ_RENT_ABSENCE_SELECTION - or raw_receipt.get("reason") != _ACS_GQ_RENT_ABSENCE_REASON - or raw_receipt.get("status") != "exact_structural_absence" + not isinstance(columns, list) + or not columns + or any(not isinstance(value, str) or not value for value in columns) + or not isinstance(dtypes, list) + or len(dtypes) != len(columns) + or any(not isinstance(value, str) or not value for value in dtypes) ): - reject(f"{label} structural-absence doctrine mismatch") - rows = nonnegative_int(raw_receipt, "rows", label=label) - unexpected = nonnegative_int( - raw_receipt, - "unexpected_null_rows", - label=label, + raise ValueError( + f"{boundary}: late PUF donor binding has malformed columns/dtypes." + ) + return + if kind == "primary_qrf_checkpoint": + require_keys( + { + *common, + "checkpoint_identity_sha256", + "checkpoint_schema_version", + "manifest_filename", + "mode", + "target_order", + "target_order_sha256", + } ) - filled = nonnegative_int( - raw_receipt, - "structural_rows_filled", - label=label, + _validate_sha256( + binding.get("checkpoint_identity_sha256"), + boundary=f"{boundary} primary-QRF checkpoint identity", ) - raw_by_role = raw_receipt.get("by_origin_role") - if not isinstance(raw_by_role, Mapping): - reject(f"{label} structural by-origin-role counts are not a mapping") - by_role: dict[str, int] = {} - for cell, count in raw_by_role.items(): - cell_channel, separator, clone_role = str(cell).partition("/clone_") - if ( - not separator - or cell_channel != ACS_STACKED_SUPPORT_CHANNEL - or not clone_role.isdigit() - or isinstance(count, bool) - or not isinstance(count, int) - or count <= 0 - ): - reject(f"{label} structural by-origin-role receipt is malformed") - by_role[str(cell)] = count - if rows != sum(by_role.values()): - reject(f"{label} structural row count does not equal its role counts") - if passed and (unexpected != 0 or filled != 0): - reject(f"{label} passing structural-absence equation is not exact") - if battery: - if raw_receipt.get("comparison_clone_index") != 0: - reject(f"{label} structural battery clone scope must be clone 0") - excluded = nonnegative_int( - raw_receipt, - "rows_excluded_from_scope", - label=label, - ) - if excluded != by_role.get(f"{ACS_STACKED_SUPPORT_CHANNEL}/clone_0", 0): - reject( - f"{label} structural battery exclusion count differs from " - "its clone-0 authority" - ) - return rows, by_role - - if gate_name == _COMPLETENESS_GATE_NAME: - expected_labels = { - f"{entity}/{family}/{column}" - for entity, family, column, _clone_index in expected_keys + expected = { + "mode": "identity_bound_checkpoint", + "checkpoint_schema_version": PRIMARY_QRF_CHECKPOINT_SCHEMA_VERSION, + "manifest_filename": PRIMARY_QRF_MANIFEST_FILENAME, + "target_order": list(PRIMARY_QRF_TARGET_ORDER), + "target_order_sha256": PRIMARY_QRF_TARGET_ORDER_SHA256, } - direction_by_label = { + if any(binding.get(key) != value for key, value in expected.items()): + raise ValueError( + f"{boundary}: late primary-QRF checkpoint semantics changed." + ) + return + if kind == "acs_pums_earnings_universe_execution_config": + require_keys( + { + *common, + "runtime_identity_owner", + "ordered_mapped_columns", + "person_scope_mode", + "contract_identity", + } + ) + if binding.get("ordered_mapped_columns") != list( + ACS_PUMS_EARNINGS_SOURCE_COLUMNS + ): + raise ValueError( + f"{boundary}: late ACS earnings-universe target order changed." + ) + if binding.get("person_scope_mode") != "whole_frame_acs_channel": + raise ValueError( + f"{boundary}: late ACS earnings-universe scope mode changed." + ) + if ( + binding.get("contract_identity") + != _CANONICAL_ACS_EARNINGS_UNIVERSE_CONTRACT_IDENTITY + ): + raise ValueError( + f"{boundary}: late ACS earnings-universe contract changed." + ) + if binding.get("runtime_identity_owner") != ( + "microcosm.build.us_runtime.acs_income_universe" + ): + raise ValueError( + f"{boundary}: late ACS earnings-universe runtime owner changed." + ) + return + if kind == "primary_puf_execution_config": + require_keys( + { + *common, + "clone_attachment", + "qrf", + "doctrines", + "capital_gains_tail", + "audit_sinks", + } + ) + clone = binding.get("clone_attachment") + qrf = binding.get("qrf") + doctrines = binding.get("doctrines") + tail = binding.get("capital_gains_tail") + audit_sinks = binding.get("audit_sinks") + if not isinstance(clone, Mapping) or set(clone) != { + "fraction", + "seed", + "support_channels", + "puf_clone_index", + }: + raise ValueError(f"{boundary}: late clone-attachment config is malformed.") + fraction = clone.get("fraction") + if ( + isinstance(fraction, bool) + or not isinstance(fraction, (int, float)) + or not np.isfinite(fraction) + or not 0 < float(fraction) <= 1 + ): + raise ValueError(f"{boundary}: late clone-attachment fraction is invalid.") + require_nonnegative_integer(clone.get("seed"), label="clone seed") + if ( + clone.get("support_channels") + != [ + BASE_ASEC_SUPPORT_CHANNEL, + PUF_TAX_DETAIL_SUPPORT_CHANNEL, + ] + or clone.get("puf_clone_index") != PUF_TAX_DETAIL_CLONE_INDEX + ): + raise ValueError(f"{boundary}: late clone support roles changed.") + qrf_keys = { + "seed", + "n_estimators", + "predictors", + "person_outputs", + "tax_unit_outputs", + "invocation_mode", + "tail_bound_quantiles", + "worker_execution", + } + if not isinstance(qrf, Mapping) or set(qrf) != qrf_keys: + raise ValueError(f"{boundary}: late primary-QRF config is malformed.") + require_nonnegative_integer(qrf.get("seed"), label="QRF seed") + require_positive_integer(qrf.get("n_estimators"), label="QRF n_estimators") + invocation = qrf.get("invocation_mode") + if not isinstance(invocation, Mapping) or set(invocation) != { + "predictors", + "person_outputs", + "tax_unit_outputs", + }: + raise ValueError( + f"{boundary}: late primary-QRF invocation mode is invalid." + ) + defaults = { + "predictors": list(PUF_TAX_DETAIL_DEFAULT_PREDICTORS), + "person_outputs": list(PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS), + "tax_unit_outputs": list(PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS), + } + for field, default in defaults.items(): + values = qrf.get(field) + mode = invocation.get(field) + if ( + not isinstance(values, list) + or not values + or any(not isinstance(value, str) or not value for value in values) + or mode not in {"canonical_default", "explicit"} + or (mode == "canonical_default" and values != default) + ): + raise ValueError( + f"{boundary}: late primary-QRF {field} binding is invalid." + ) + if qrf.get("tail_bound_quantiles") != ( + puf_tax_detail_tail_bound_quantiles_identity() + ): + raise ValueError(f"{boundary}: late primary-QRF tail bounds changed.") + if qrf.get("worker_execution") != ( + _late_primary_qrf_worker_execution_binding() + ): + raise ValueError(f"{boundary}: late primary-QRF worker binding changed.") + if doctrines != { + "require_complete_recipient_predictors": True, + "absent_cells": PUF_ABSENT_CELLS_PRESERVE_NULLS, + "whole_pool_output_universes": { + "person.s_corp_income": (us_puf_s_corp_universe_zero_rule_identity()), + }, + }: + raise ValueError(f"{boundary}: late primary-PUF doctrines changed.") + if audit_sinks != { + "fit_records": "enabled", + "tail_bound_diagnostics": "enabled", + "recipient_predictor_universe": "required_receipt", + }: + raise ValueError(f"{boundary}: late primary-PUF audit sinks changed.") + tail_inputs = puf_capital_gains_tail_execution_inputs_identity() + expected_tail_keys = { + "enabled", + "seed", + "support_contract", + "spec", + "soi_e19200_agi_bands", + "concentration_gate", + } + if ( + not isinstance(tail, Mapping) + or set(tail) != expected_tail_keys + or tail.get("enabled") is not True + or tail.get("seed") != qrf.get("seed") + or tail.get("support_contract") + != puf_capital_gains_tail_support_contract_identity() + or tail.get("spec") != puf_capital_gains_tail_spec_identity() + or tail.get("soi_e19200_agi_bands") != tail_inputs["soi_e19200_agi_bands"] + or tail.get("concentration_gate") != tail_inputs["concentration_gate"] + ): + raise ValueError(f"{boundary}: late capital-gains-tail config changed.") + return + if kind == "source_operator_receipt": + require_keys({*common, "source_operator", "source_receipt_sha256"}) + expected_operator = column.removeprefix("@source_receipt:") + if binding.get("source_operator") != expected_operator: + raise ValueError(f"{boundary}: late source receipt owner changed.") + _validate_sha256( + binding.get("source_receipt_sha256"), + boundary=f"{boundary} source receipt", + ) + return + if kind == "source_finalizer_execution_config": + expected = _late_source_finalizer_execution_binding(live_runtime=False) + require_keys(set(expected)) + if _json_ready(binding) != _json_ready(expected): + raise ValueError(f"{boundary}: late source-finalizer config changed.") + return + if kind == "post_clone_source_execution_config": + expected_operator = producer.removeprefix("source:") + if ( + producer != f"source:{expected_operator}" + or binding.get("operator") != expected_operator + ): + raise ValueError(f"{boundary}: late source execution owner changed.") + expected = _late_source_execution_config_binding( + producer, + live_runtime=False, + ) + require_keys(set(expected)) + if _json_ready(binding) != _json_ready(expected): + raise ValueError(f"{boundary}: late source execution config changed.") + return + if kind == "late_transfer_model_config": + require_keys( + { + *common, + "producer", + "entity", + "family", + "ordered_targets", + "seed", + "n_estimators", + "max_targets_per_fit", + "donor_spine", + "donor_channel", + "donor_selection", + "donor_projection", + "transfer_execution_contract", + } + ) + group = next( + ( + item + for item in CANONICAL_US_LATE_TRANSFER_GROUPS + if item.name == producer + ), + None, + ) + if group is None or any( + binding.get(key) != value + for key, value in { + "producer": group.name, + "entity": group.entity, + "family": group.family, + "ordered_targets": list(group.targets), + "donor_spine": ASEC_PUF_DONOR_SPINE, + "donor_channel": None, + "donor_selection": ("all_rows_from_post_puf_asec_origin_projection"), + "donor_projection": { + "support_channel": BASE_ASEC_SUPPORT_CHANNEL, + "support_clone_index": PUF_TAX_DETAIL_CLONE_INDEX, + }, + }.items() + ): + raise ValueError(f"{boundary}: late transfer model owner/targets changed.") + require_nonnegative_integer(binding.get("seed"), label="transfer seed") + require_positive_integer( + binding.get("n_estimators"), label="transfer n_estimators" + ) + if ( + require_positive_integer( + binding.get("max_targets_per_fit"), + label="transfer max_targets_per_fit", + ) + != DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT + ): + raise ValueError( + f"{boundary}: late transfer max_targets_per_fit is noncanonical." + ) + if binding.get("transfer_execution_contract") != ( + acs_transfer_runtime.acs_transfer_execution_contract_identity( + targets=group.targets, + derive_schedule_d=False, + ) + ): + raise ValueError(f"{boundary}: late transfer execution contract changed.") + return + if kind == "late_transfer_target_bank": + mode = binding.get("mode") + if mode == "ephemeral_no_checkpoint": + require_keys({*common, "mode"}) + return + if mode == "identity_bound_checkpoint": + require_keys({*common, "mode", "identity_sha256"}) + _validate_sha256( + binding.get("identity_sha256"), + boundary=f"{boundary} transfer target-bank identity", + ) + return + raise ValueError(f"{boundary}: late transfer target-bank mode is invalid.") + raise AssertionError(f"Unhandled late virtual resource kind {kind!r}.") + + +def _late_available_input_receipt( + *, + producer: str, + entity: str, + column: str, + rows: int, + binding: Mapping[str, object], +) -> dict[str, object]: + """Create one exact, hash-bound virtual-resource availability receipt.""" + + if isinstance(rows, bool) or not isinstance(rows, int) or rows <= 0: + raise ValueError( + "US late-producer virtual-resource rows must be a positive integer; " + f"got {rows!r}." + ) + normalized_binding = _json_ready(binding) + expected_kind = _late_virtual_resource_kind(column) + if normalized_binding.get("resource_kind") != expected_kind: + raise ValueError( + f"US late-producer resource {entity}.{column} requires " + f"resource_kind={expected_kind!r}." + ) + expected_schema_version = _late_resource_binding_schema_version(column) + if normalized_binding.get("schema_version") != expected_schema_version: + raise ValueError( + f"US late-producer resource {entity}.{column} requires binding " + f"schema_version={expected_schema_version}." + ) + receipt = { + "receipt_id": f"available_input:{producer}:{entity}.{column}", + "status": "available", + "producer": producer, + "entity": entity, + "column": column, + "rows": rows, + "binding": normalized_binding, + "binding_sha256": _canonical_sha256(normalized_binding), + } + _validate_late_available_input_receipt( + receipt, + producer=producer, + entity=entity, + column=column, + boundary="US late-producer resource construction", + ) + return receipt + + +def _validate_late_available_input_receipt( + receipt: object, + *, + producer: str, + entity: str, + column: str, + boundary: str, +) -> None: + """Validate one virtual input receipt without trusting a row count alone.""" + + expected_keys = { + "receipt_id", + "status", + "producer", + "entity", + "column", + "rows", + "binding", + "binding_sha256", + } + if not isinstance(receipt, Mapping) or set(receipt) != expected_keys: + raise ValueError( + f"{boundary}: late-producer available-input receipt " + f"{entity}.{column!s} does not carry its exact semantic binding." + ) + expected = { + "receipt_id": f"available_input:{producer}:{entity}.{column}", + "status": "available", + "producer": producer, + "entity": entity, + "column": column, + } + if any(receipt.get(key) != value for key, value in expected.items()): + raise ValueError( + f"{boundary}: late-producer available-input receipt " + f"{entity}.{column!s} is misbound." + ) + rows = receipt.get("rows") + if isinstance(rows, bool) or not isinstance(rows, int) or rows <= 0: + raise ValueError( + f"{boundary}: late-producer available-input receipt " + f"{entity}.{column!s} has invalid rows={rows!r}." + ) + binding = receipt.get("binding") + expected_kind = _late_virtual_resource_kind(column) + expected_schema_version = _late_resource_binding_schema_version(column) + if ( + not isinstance(binding, Mapping) + or binding.get("resource_kind") != expected_kind + or binding.get("schema_version") != expected_schema_version + ): + raise ValueError( + f"{boundary}: late-producer available-input receipt " + f"{entity}.{column!s} has a malformed semantic binding." + ) + binding_sha256 = receipt.get("binding_sha256") + _validate_sha256( + binding_sha256, + boundary=f"{boundary} late resource {entity}.{column} binding", + ) + if binding_sha256 != _canonical_sha256(_json_ready(binding)): + raise ValueError( + f"{boundary}: late-producer available-input receipt " + f"{entity}.{column!s} binding SHA-256 mismatch." + ) + _validate_late_resource_binding( + binding, + producer=producer, + entity=entity, + column=column, + boundary=boundary, + ) + + +def _late_available_input_receipt_is_valid( + receipt: object, + *, + producer: str, + entity: str, + column: str, +) -> bool: + try: + _validate_late_available_input_receipt( + receipt, + producer=producer, + entity=entity, + column=column, + boundary="US late-producer readiness", + ) + except (TypeError, ValueError): + return False + return True + + +def _late_string_sequence( + values: Sequence[str] | None, + *, + label: str, +) -> list[str] | None: + if values is None: + return None + if isinstance(values, (str, bytes)) or any( + not isinstance(value, str) or not value for value in values + ): + raise TypeError(f"{label} must be a sequence of non-empty strings or None.") + return list(values) + + +_PRIMARY_QRF_WORKER_MODULE = "microcosm.build.us_runtime.puf_qrf_worker" +_PRIMARY_QRF_SEMANTIC_ENVIRONMENT_NAMES = ( + "POPULACE_FIT_N_JOBS", + "POPULACE_FIT_PREDICT_WORKERS", +) + + +def _late_primary_qrf_worker_execution_binding() -> dict[str, object]: + """Bind the interpreter and reviewed inherited environment controls.""" + + fit_jobs_raw = os.environ.get("POPULACE_FIT_N_JOBS") + if fit_jobs_raw is None: + fit_jobs = -1 + else: + try: + fit_jobs = int(fit_jobs_raw) + except ValueError as exc: + raise ValueError( + "POPULACE_FIT_N_JOBS must be a positive integer for the " + "primary-QRF worker binding." + ) from exc + if fit_jobs < 1 or str(fit_jobs) != fit_jobs_raw: + raise ValueError( + "POPULACE_FIT_N_JOBS must be a canonical positive integer for " + "the primary-QRF worker binding." + ) + predict_workers_raw = os.environ.get("POPULACE_FIT_PREDICT_WORKERS") + if predict_workers_raw is None or not predict_workers_raw.strip(): + predict_workers = os.cpu_count() or 1 + predict_workers_source = "os_cpu_count_fallback" + else: + try: + predict_workers = int(predict_workers_raw) + except ValueError as exc: + raise ValueError( + "POPULACE_FIT_PREDICT_WORKERS must be a positive integer for " + "the primary-QRF worker binding." + ) from exc + if predict_workers < 1: + raise ValueError( + "POPULACE_FIT_PREDICT_WORKERS must be positive for the " + "primary-QRF worker binding." + ) + predict_workers_source = "environment_override" + executable = Path(sys.executable) + return { + "module": _PRIMARY_QRF_WORKER_MODULE, + "argv_template": [ + str(executable), + "-m", + _PRIMARY_QRF_WORKER_MODULE, + "--checkpoint-dir", + "{checkpoint_dir}", + "--target-index", + "{target_index}", + ], + "interpreter": { + "executable": str(executable), + "resolved_executable": str(executable.resolve()), + "implementation": sys.implementation.name, + "cache_tag": sys.implementation.cache_tag, + "version": list(sys.version_info[:3]), + }, + "environment": { + "policy": "inherit_parent_environment_with_bound_fit_controls", + "overrides": {}, + "semantic_controls": { + "POPULACE_FIT_N_JOBS": { + "configured": fit_jobs_raw, + "resolved": fit_jobs, + }, + "POPULACE_FIT_PREDICT_WORKERS": { + "configured": predict_workers_raw, + "resolved": predict_workers, + "resolution": predict_workers_source, + }, + }, + "bound_names": list(_PRIMARY_QRF_SEMANTIC_ENVIRONMENT_NAMES), + }, + } + + +def _late_primary_execution_config_binding( + *, + clone_attachment_fraction: float, + clone_attachment_seed: int, + seed: int, + n_estimators: int, + predictors: Sequence[str] | None, + person_outputs: Sequence[str] | None, + tax_unit_outputs: Sequence[str] | None, + fit_records_enabled: bool, + tail_bound_diagnostics_enabled: bool, + capital_gains_tail_spec: PufAggregateDisaggregationSpec | None = None, + capital_gains_tail_agi_bands: Sequence[PufE19200AgiBand] | None = None, +) -> dict[str, object]: + """Return every non-Frame control consumed by the primary callback.""" + + resolved_predictors = _late_string_sequence( + PUF_TAX_DETAIL_DEFAULT_PREDICTORS if predictors is None else predictors, + label="predictors", + ) + resolved_person_outputs = _late_string_sequence( + PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS + if person_outputs is None + else person_outputs, + label="person_outputs", + ) + resolved_tax_unit_outputs = _late_string_sequence( + PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS + if tax_unit_outputs is None + else tax_unit_outputs, + label="tax_unit_outputs", + ) + tail_inputs = puf_capital_gains_tail_execution_inputs_identity( + spec=capital_gains_tail_spec, + agi_bands=capital_gains_tail_agi_bands, + ) + return { + "resource_kind": "primary_puf_execution_config", + "schema_version": 4, + "clone_attachment": { + "fraction": float(clone_attachment_fraction), + "seed": clone_attachment_seed, + "support_channels": [ + BASE_ASEC_SUPPORT_CHANNEL, + PUF_TAX_DETAIL_SUPPORT_CHANNEL, + ], + "puf_clone_index": PUF_TAX_DETAIL_CLONE_INDEX, + }, + "qrf": { + "seed": seed, + "n_estimators": n_estimators, + "predictors": resolved_predictors, + "person_outputs": resolved_person_outputs, + "tax_unit_outputs": resolved_tax_unit_outputs, + "invocation_mode": { + "predictors": ( + "canonical_default" if predictors is None else "explicit" + ), + "person_outputs": ( + "canonical_default" if person_outputs is None else "explicit" + ), + "tax_unit_outputs": ( + "canonical_default" if tax_unit_outputs is None else "explicit" + ), + }, + "tail_bound_quantiles": (puf_tax_detail_tail_bound_quantiles_identity()), + "worker_execution": _late_primary_qrf_worker_execution_binding(), + }, + "doctrines": { + "require_complete_recipient_predictors": True, + "absent_cells": PUF_ABSENT_CELLS_PRESERVE_NULLS, + "whole_pool_output_universes": { + "person.s_corp_income": (us_puf_s_corp_universe_zero_rule_identity()), + }, + }, + "capital_gains_tail": { + "enabled": True, + "seed": seed, + "support_contract": puf_capital_gains_tail_support_contract_identity( + capital_gains_tail_agi_bands + ), + "spec": tail_inputs["aggregate_disaggregation_spec"], + "soi_e19200_agi_bands": tail_inputs["soi_e19200_agi_bands"], + "concentration_gate": tail_inputs["concentration_gate"], + }, + "audit_sinks": { + "fit_records": "enabled" if fit_records_enabled else "disabled", + "tail_bound_diagnostics": ( + "enabled" if tail_bound_diagnostics_enabled else "disabled" + ), + "recipient_predictor_universe": "required_receipt", + }, + } + + +def _late_puf_donor_resource_semantics_binding() -> dict[str, object]: + """Describe the dynamic donor binding without embedding build-specific bytes.""" + + return { + "resource_kind": "puf_donor_tax_units", + "schema_version": 1, + "runtime_identity": { + "codec": _LATE_TABLE_DIGEST_CODEC, + "fields": ["table_content_sha256", "ordered_columns", "dtypes"], + "normalization": "canonical_table_string_dtypes", + }, + "source_input_pins": ["processed_puf", "puf_source_year"], + } + + +def _late_primary_qrf_checkpoint_static_binding() -> dict[str, object]: + """Return the non-recursive primary-QRF bank semantics.""" + + return { + "resource_kind": "primary_qrf_checkpoint", + "schema_version": 1, + "mode": "identity_bound_checkpoint", + "checkpoint_schema_version": PRIMARY_QRF_CHECKPOINT_SCHEMA_VERSION, + "manifest_filename": PRIMARY_QRF_MANIFEST_FILENAME, + "target_order": list(PRIMARY_QRF_TARGET_ORDER), + "target_order_sha256": PRIMARY_QRF_TARGET_ORDER_SHA256, + } + + +def stacked_late_primary_resource_receipts( + donor_tax_units: pd.DataFrame, + *, + primary_qrf_checkpoint_identity_sha256: str, + clone_attachment_fraction: float, + clone_attachment_seed: int, + seed: int, + n_estimators: int, + fit_records_enabled: bool, + tail_bound_diagnostics_enabled: bool, + predictors: Sequence[str] | None = None, + person_outputs: Sequence[str] | None = None, + tax_unit_outputs: Sequence[str] | None = None, + capital_gains_tail_spec: PufAggregateDisaggregationSpec | None = None, + capital_gains_tail_agi_bands: Sequence[PufE19200AgiBand] | None = None, +) -> dict[str, dict[str, object]]: + """Bind every non-Frame input consumed by the primary PUF producer.""" + + if not isinstance(donor_tax_units, pd.DataFrame) or donor_tax_units.empty: + raise ValueError("US late primary-PUF donor must be a nonempty DataFrame.") + _validate_sha256( + primary_qrf_checkpoint_identity_sha256, + boundary="US late primary-QRF checkpoint identity", + ) + if ( + isinstance(clone_attachment_fraction, bool) + or not isinstance(clone_attachment_fraction, (int, float)) + or not np.isfinite(clone_attachment_fraction) + or not 0 < float(clone_attachment_fraction) <= 1 + ): + raise ValueError( + "US late primary-PUF clone attachment fraction must be finite in " + f"(0, 1]; got {clone_attachment_fraction!r}." + ) + for label, value in { + "clone_attachment_seed": clone_attachment_seed, + "seed": seed, + }.items(): + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise ValueError(f"US late primary-PUF {label} must be non-negative.") + if ( + isinstance(n_estimators, bool) + or not isinstance(n_estimators, int) + or n_estimators <= 0 + ): + raise ValueError("US late primary-PUF n_estimators must be positive.") + if not isinstance(fit_records_enabled, bool) or not isinstance( + tail_bound_diagnostics_enabled, bool + ): + raise TypeError("US late primary-PUF audit-sink modes must be booleans.") + normalized_donor = canonicalize_table_string_dtypes( + donor_tax_units, + boundary="late primary-PUF donor resource binding", + table_name="puf_donor_tax_units", + ) + donor_binding = { + "resource_kind": "puf_donor_tax_units", + "schema_version": 1, + "table_content_sha256": _late_table_values_sha256( + normalized_donor, + ), + "ordered_columns": [str(column) for column in normalized_donor.columns], + "dtypes": [str(dtype) for dtype in normalized_donor.dtypes], + } + checkpoint_binding = { + **_late_primary_qrf_checkpoint_static_binding(), + "checkpoint_identity_sha256": primary_qrf_checkpoint_identity_sha256, + } + config_binding = _late_primary_execution_config_binding( + clone_attachment_fraction=clone_attachment_fraction, + clone_attachment_seed=clone_attachment_seed, + seed=seed, + n_estimators=n_estimators, + predictors=predictors, + person_outputs=person_outputs, + tax_unit_outputs=tax_unit_outputs, + fit_records_enabled=fit_records_enabled, + tail_bound_diagnostics_enabled=tail_bound_diagnostics_enabled, + capital_gains_tail_spec=capital_gains_tail_spec, + capital_gains_tail_agi_bands=capital_gains_tail_agi_bands, + ) + return { + "tax_unit.@puf_donor_tax_units": _late_available_input_receipt( + producer=US_LATE_PRIMARY_PUF_STAGE, + entity="tax_unit", + column="@puf_donor_tax_units", + rows=int(len(donor_tax_units)), + binding=donor_binding, + ), + "tax_unit.@primary_qrf_checkpoint": _late_available_input_receipt( + producer=US_LATE_PRIMARY_PUF_STAGE, + entity="tax_unit", + column="@primary_qrf_checkpoint", + rows=1, + binding=checkpoint_binding, + ), + f"tax_unit.{US_LATE_PRIMARY_EXECUTION_CONFIG_INPUT}": ( + _late_available_input_receipt( + producer=US_LATE_PRIMARY_PUF_STAGE, + entity="tax_unit", + column=US_LATE_PRIMARY_EXECUTION_CONFIG_INPUT, + rows=1, + binding=config_binding, + ) + ), + } + + +def _validate_stacked_late_primary_checkpoint_input_binding( + binding: object, + *, + boundary: str, +) -> None: + """Authenticate the sidecar that prevents stale primary-QRF bank reuse.""" + + expected_keys = { + "artifact_kind", + "schema_version", + "primary_resource_receipts", + "sha256", + } + if not isinstance(binding, Mapping) or set(binding) != expected_keys: + raise ValueError(f"{boundary}: primary-QRF input binding schema drifted.") + if ( + binding.get("artifact_kind") != _LATE_PRIMARY_QRF_INPUT_BINDING_ARTIFACT_KIND + or binding.get("schema_version") != 1 + ): + raise ValueError(f"{boundary}: primary-QRF input binding identity changed.") + resources = binding.get("primary_resource_receipts") + contract = CANONICAL_US_LATE_PRODUCER_REGISTRY[US_LATE_PRIMARY_PUF_STAGE] + expected_resources = _late_contract_available_input_keys(contract) + if not isinstance(resources, Mapping) or set(resources) != expected_resources: + raise ValueError( + f"{boundary}: primary-QRF input binding does not cover its exact " + "declared resource surface." + ) + for key, receipt in resources.items(): + entity, column = key.split(".", 1) + _validate_late_available_input_receipt( + receipt, + producer=US_LATE_PRIMARY_PUF_STAGE, + entity=entity, + column=column, + boundary=boundary, + ) + unsigned = dict(binding) + sha256 = unsigned.pop("sha256") + _validate_sha256(sha256, boundary=f"{boundary} primary-QRF input binding") + if sha256 != _canonical_sha256(_json_ready(unsigned)): + raise ValueError(f"{boundary}: primary-QRF input binding SHA-256 mismatch.") + + +def stacked_late_primary_checkpoint_input_binding( + primary_resource_receipts: Mapping[str, Mapping[str, object]], +) -> dict[str, object]: + """Build the durable input sidecar for the primary-QRF checkpoint bank.""" + + resources = { + key: _json_ready(receipt) + for key, receipt in sorted(primary_resource_receipts.items()) + } + payload: dict[str, object] = { + "artifact_kind": _LATE_PRIMARY_QRF_INPUT_BINDING_ARTIFACT_KIND, + "schema_version": 1, + "primary_resource_receipts": resources, + } + payload["sha256"] = _canonical_sha256(payload) + _validate_stacked_late_primary_checkpoint_input_binding( + payload, + boundary="US stacked late primary-QRF input-binding construction", + ) + return payload + + +_SOURCE_MANIFEST_STAGE_BY_OPERATOR: Mapping[str, str] = MappingProxyType( + { + "with_us_prior_year_income_inputs": "prior_year_income", + "with_us_medicare_take_up_input": "medicare_take_up_input", + "with_us_pregnancy_inputs": "pregnancy", + "with_us_wic_claim_input": "wic_claim_input", + "with_us_child_support_inputs": "child_support_inputs", + "with_us_disability_benefits": "disability_benefits_input", + "with_us_workers_compensation": "workers_compensation_input", + "with_us_weeks_unemployed": "weeks_unemployed_input", + "with_us_childcare_inputs": "childcare_inputs", + "with_us_adult_care_inputs": "adult_care_inputs", + "with_us_energy_subsidy_input": "energy_subsidy", + "with_us_retirement_contribution_inputs": "retirement_contributions", + "with_us_retirement_distribution_inputs": "retirement_distributions", + "with_us_immigration_inputs": "immigration_status", + "with_us_education_inputs": "education_inputs", + } +) +_SOURCE_STAGE_SPEC_RESOLVER_BY_OPERATOR: Mapping[str, tuple[str, str]] = ( + MappingProxyType( + { + "with_us_prior_year_income_inputs": ( + "microcosm.build.us_runtime.prior_year_income", + "us_prior_year_income_stage_spec", + ), + "with_us_medicare_take_up_input": ( + "microcosm.build.us_runtime.medicare_take_up", + "us_medicare_take_up_stage_spec", + ), + "with_us_pregnancy_inputs": ( + "microcosm.build.us_runtime.pregnancy", + "us_pregnancy_stage_spec", + ), + "with_us_wic_claim_input": ( + "microcosm.build.us_runtime.wic_claim", + "us_wic_claim_stage_spec", + ), + "with_us_child_support_inputs": ( + "microcosm.build.us_runtime.child_support", + "us_child_support_stage_spec", + ), + "with_us_disability_benefits": ( + "microcosm.build.us_runtime.disability_benefits", + "us_disability_benefits_stage_spec", + ), + "with_us_workers_compensation": ( + "microcosm.build.us_runtime.workers_compensation", + "us_workers_compensation_stage_spec", + ), + "with_us_weeks_unemployed": ( + "microcosm.build.us_runtime.weeks_unemployed", + "us_weeks_unemployed_stage_spec", + ), + "with_us_childcare_inputs": ( + "microcosm.build.us_runtime.childcare", + "us_childcare_stage_spec", + ), + "with_us_adult_care_inputs": ( + "microcosm.build.us_runtime.adult_care", + "us_adult_care_stage_spec", + ), + "with_us_energy_subsidy_input": ( + "microcosm.build.us_runtime.energy_subsidy", + "us_energy_subsidy_stage_spec", + ), + "with_us_retirement_contribution_inputs": ( + "microcosm.build.us_runtime.retirement_contributions", + "us_retirement_contributions_stage_spec", + ), + "with_us_retirement_distribution_inputs": ( + "microcosm.build.us_runtime.retirement_distributions", + "us_retirement_distributions_stage_spec", + ), + "with_us_immigration_inputs": ( + "microcosm.build.us_runtime.immigration", + "us_immigration_stage_spec", + ), + "with_us_education_inputs": ( + "microcosm.build.us_runtime.education_inputs", + "us_education_inputs_stage_spec", + ), + } + ) +) +_DIRECT_HOUSING_ASSISTANCE_SOURCE_OPERATOR = ( + "impute_us_housing_assistance_to_puf_support" +) +if len(_SOURCE_MANIFEST_STAGE_BY_OPERATOR) != 15 or set( + _SOURCE_MANIFEST_STAGE_BY_OPERATOR +) | {_DIRECT_HOUSING_ASSISTANCE_SOURCE_OPERATOR} != set( + POOL_POST_CLONE_SOURCE_OPERATOR_ORDER +): + raise RuntimeError( + "US late source callback-identity map must cover exactly fifteen " + "manifest stages plus the direct housing-assistance QRF." + ) +if set(_SOURCE_STAGE_SPEC_RESOLVER_BY_OPERATOR) != set( + _SOURCE_MANIFEST_STAGE_BY_OPERATOR +): + raise RuntimeError( + "US late source runtime-spec resolver map must cover exactly the " + "fifteen manifest-backed callbacks." + ) + + +def _late_source_stage_spec_binding( + operator: str, + *, + resource: object | None = None, +) -> dict[str, object] | None: + """Resolve the exact packaged SourceStageSpec consumed by a callback.""" + + if operator == _DIRECT_HOUSING_ASSISTANCE_SOURCE_OPERATOR: + return None + try: + stage_name = _SOURCE_MANIFEST_STAGE_BY_OPERATOR[operator] + except KeyError as exc: + raise ValueError( + f"US late source operator {operator!r} has no manifest-stage binding." + ) from exc + resolved_resource = ( + files("microcosm.build.us").joinpath("source_stages.json") + if resource is None + else resource + ) + if not hasattr(resolved_resource, "read_bytes"): + raise TypeError("US source-stage binding resource must expose read_bytes().") + manifest = load_source_manifest(resolved_resource) + stage_map = manifest.stage_map() + if stage_name not in stage_map: + raise ValueError( + f"US source manifest declares no bound stage {stage_name!r} for " + f"operator {operator!r}." + ) + stage_spec = _json_ready(asdict(stage_map[stage_name])) + resolver_module, resolver_name = _SOURCE_STAGE_SPEC_RESOLVER_BY_OPERATOR[operator] + runtime_spec_verified = resource is None + if runtime_spec_verified: + runtime_module = importlib.import_module(resolver_module) + runtime_resolver = getattr(runtime_module, resolver_name, None) + if not callable(runtime_resolver): + raise RuntimeError( + f"US late source runtime resolver {resolver_module}.{resolver_name} " + "is unavailable." + ) + runtime_stage_spec = _json_ready(asdict(runtime_resolver())) + if runtime_stage_spec != stage_spec: + raise ValueError( + f"US late source operator {operator!r} runtime SourceStageSpec " + "differs from the content-bound packaged manifest." + ) + asset_bytes = resolved_resource.read_bytes() + return { + "asset": "microcosm.build.us/source_stages.json", + "asset_sha256": hashlib.sha256(asset_bytes).hexdigest(), + "manifest": { + "country": manifest.country, + "version": manifest.version, + "policy": manifest.policy, + }, + "stage_name": stage_name, + "resolved_stage_spec": stage_spec, + "resolved_stage_spec_sha256": _canonical_sha256(stage_spec), + "runtime_stage_spec_resolver": { + "module": resolver_module, + "callable": resolver_name, + }, + "runtime_stage_spec_verified": runtime_spec_verified, + } + + +def _late_source_execution_config_binding( + producer_name: str, + *, + live_runtime: bool = True, +) -> dict[str, object]: + """Return all fixed controls consumed by one post-clone source callback.""" + + operator = producer_name.removeprefix("source:") + if producer_name != f"source:{operator}": + raise ValueError( + f"US late source producer name is malformed: {producer_name!r}." + ) + if live_runtime: + seed = multispine_pool_runtime.POOL_RANDOM_SEED + time_period = multispine_pool_runtime.POOL_TIME_PERIOD + allow_existing = ( + multispine_pool_runtime.POOL_SOURCE_ALLOW_EXISTING_WITHOUT_SOURCE + ) + housing_n_estimators = ( + multispine_pool_runtime.POOL_HOUSING_ASSISTANCE_N_ESTIMATORS + ) + housing_max_train_samples = ( + multispine_pool_runtime.POOL_HOUSING_ASSISTANCE_MAX_TRAIN_SAMPLES + ) + operator_registry = ( + multispine_pool_runtime.POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + ) + phase = multispine_pool_runtime._POST_CLONE_PHASE + operator_contracts = multispine_pool_runtime.POOL_OPERATOR_CONTRACTS + output_families = multispine_pool_runtime.PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES + formula_owned_outputs = multispine_pool_runtime._FORMULA_OWNED_SOURCE_OUTPUTS + else: + seed = POOL_RANDOM_SEED + time_period = POOL_TIME_PERIOD + allow_existing = POOL_SOURCE_ALLOW_EXISTING_WITHOUT_SOURCE + housing_n_estimators = POOL_HOUSING_ASSISTANCE_N_ESTIMATORS + housing_max_train_samples = POOL_HOUSING_ASSISTANCE_MAX_TRAIN_SAMPLES + operator_registry = POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + phase = POOL_POST_CLONE_SOURCE_PHASE + operator_contracts = POOL_OPERATOR_CONTRACTS + output_families = PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES + formula_owned_outputs = FORMULA_OWNED_SOURCE_COLUMNS + try: + operator_contract = operator_contracts[operator] + except KeyError as exc: + raise ValueError( + f"US late source runtime declares no operator {operator!r}." + ) from exc + family_outputs = output_families[operator_contract.family] + return { + "resource_kind": "post_clone_source_execution_config", + "schema_version": 3, + "operator": operator, + "phase": phase, + "operator_registry": list(operator_registry), + "operator_contract": { + "family": operator_contract.family, + "phases": list(operator_contract.phases), + "mechanism": operator_contract.mechanism, + "execution_scope": operator_contract.execution_scope, + }, + "declared_output_family": { + entity: sorted(columns) + for entity, columns in sorted(family_outputs.items()) + }, + "formula_owned_outputs_removed": { + entity: sorted(set(columns) & set(formula_owned_outputs.get(entity, ()))) + for entity, columns in sorted(family_outputs.items()) + if set(columns) & set(formula_owned_outputs.get(entity, ())) + }, + "seed": seed, + "time_period": ( + None + if operator == "impute_us_housing_assistance_to_puf_support" + else time_period + ), + "force_puf_imputation": ( + True if operator == "with_us_retirement_distribution_inputs" else None + ), + "allow_existing_without_source": ( + allow_existing + if operator + in { + "with_us_child_support_inputs", + "with_us_disability_benefits", + "with_us_workers_compensation", + "with_us_childcare_inputs", + "with_us_adult_care_inputs", + "with_us_energy_subsidy_input", + } + else None + ), + "housing_assistance_qrf": ( + { + "n_estimators": housing_n_estimators, + "max_train_samples": housing_max_train_samples, + } + if operator == _DIRECT_HOUSING_ASSISTANCE_SOURCE_OPERATOR + else None + ), + "external_sidecars": ( + {"asec_2023_source": {"mode": "not_supplied"}} + if operator == "with_us_weeks_unemployed" + else {"asec_education_source": {"mode": "not_supplied"}} + if operator == "with_us_education_inputs" + else {} + ), + "source_stage_spec": _late_source_stage_spec_binding(operator), + } + + +def _late_source_resource_receipts( + *, + producer_name: str, +) -> dict[str, dict[str, object]]: + """Bind the fixed controls consumed by one post-clone source callback.""" + + binding = _late_source_execution_config_binding(producer_name) + return { + f"person.{US_LATE_SOURCE_EXECUTION_CONFIG_INPUT}": ( + _late_available_input_receipt( + producer=producer_name, + entity="person", + column=US_LATE_SOURCE_EXECUTION_CONFIG_INPUT, + rows=1, + binding=binding, + ) + ) + } + + +def _late_source_finalizer_execution_binding( + *, + live_runtime: bool = True, +) -> dict[str, object]: + """Bind every doctrine input consumed by the source finalizer callback.""" + + if live_runtime: + phase = multispine_pool_runtime._POST_CLONE_PHASE + operator_registry = ( + multispine_pool_runtime.POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + ) + formula_owned_outputs = multispine_pool_runtime._FORMULA_OWNED_SOURCE_OUTPUTS + deferred_inputs = multispine_pool_runtime.POOL_DEFERRED_TRANSFER_INPUTS + deferred_status = multispine_pool_runtime.POOL_DEFERRED_TRANSFER_STATUS + else: + phase = POOL_POST_CLONE_SOURCE_PHASE + operator_registry = POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + formula_owned_outputs = FORMULA_OWNED_SOURCE_COLUMNS + deferred_inputs = POOL_DEFERRED_TRANSFER_INPUTS + deferred_status = POOL_DEFERRED_TRANSFER_STATUS + return { + "resource_kind": "source_finalizer_execution_config", + "schema_version": 2, + "phase": phase, + "source_operator_registry": list(operator_registry), + "formula_owned_output_exclusions": { + entity: sorted(columns) + for entity, columns in sorted(formula_owned_outputs.items()) + }, + "deferred_transfer_inputs": _json_ready(deferred_inputs), + "deferred_status": deferred_status, + } + + +def _late_source_finalizer_resource_receipts() -> dict[str, dict[str, object]]: + """Bind the finalizer's phase, registry, exclusions, and deferral contract.""" + + column = US_LATE_SOURCE_FINALIZER_EXECUTION_CONFIG_INPUT + return { + f"person.{column}": _late_available_input_receipt( + producer=US_LATE_SOURCE_FINALIZER_STAGE, + entity="person", + column=column, + rows=1, + binding=_late_source_finalizer_execution_binding(), + ) + } + + +def _assert_late_callback_consumed_bound_config( + *, + producer: str, + entity: str, + column: str, + available_input_receipts: Mapping[str, Mapping[str, object]], + actual_binding: Mapping[str, object], +) -> None: + """Refuse a callback whose live runtime config differs from its DAG input.""" + + key = f"{entity}.{column}" + receipt = available_input_receipts.get(key) + bound = receipt.get("binding") if isinstance(receipt, Mapping) else None + if not isinstance(bound, Mapping) or _json_ready(bound) != _json_ready( + actual_binding + ): + raise ValueError( + f"Late producer {producer!r} consumed a runtime execution config " + f"that differs from its bound input {key}." + ) + + +_CANONICAL_ACS_EARNINGS_UNIVERSE_CONTRACT_IDENTITY = dict( + acs_pums_earnings_universe_contract_identity() +) + + +def _late_acs_earnings_universe_execution_binding() -> dict[str, object]: + """Return the exact rules and scope consumed by the universe producer.""" + + return { + "resource_kind": "acs_pums_earnings_universe_execution_config", + "schema_version": 2, + "runtime_identity_owner": ("microcosm.build.us_runtime.acs_income_universe"), + "ordered_mapped_columns": list( + acs_income_universe_runtime.ACS_PUMS_EARNINGS_SOURCE_COLUMNS + ), + "person_scope_mode": "whole_frame_acs_channel", + "contract_identity": ( + acs_income_universe_runtime.acs_pums_earnings_universe_contract_identity() + ), + } + + +def _late_acs_earnings_universe_resource_receipts() -> dict[str, dict[str, object]]: + """Bind the exact rules and scope consumed by the universe producer.""" + + column = US_LATE_ACS_EARNINGS_UNIVERSE_CONFIG_INPUT + return { + f"person.{column}": _late_available_input_receipt( + producer=US_LATE_ACS_EARNINGS_UNIVERSE_STAGE, + entity="person", + column=column, + rows=1, + binding=_late_acs_earnings_universe_execution_binding(), + ) + } + + +def _late_transfer_model_config_binding( + *, + group_name: str, + entity: str, + family: str, + targets: Sequence[str], + seed: int, + n_estimators: int, + max_targets_per_fit: int, +) -> dict[str, object]: + """Return every static model and donor-selection control for one group.""" + + return { + "resource_kind": "late_transfer_model_config", + "schema_version": 3, + "producer": group_name, + "entity": entity, + "family": family, + "ordered_targets": list(targets), + "seed": seed, + "n_estimators": n_estimators, + "max_targets_per_fit": max_targets_per_fit, + "donor_spine": ASEC_PUF_DONOR_SPINE, + "donor_channel": None, + "donor_selection": "all_rows_from_post_puf_asec_origin_projection", + "donor_projection": { + "support_channel": BASE_ASEC_SUPPORT_CHANNEL, + "support_clone_index": PUF_TAX_DETAIL_CLONE_INDEX, + }, + "transfer_execution_contract": ( + acs_transfer_runtime.acs_transfer_execution_contract_identity( + targets=targets, + derive_schedule_d=False, + ) + ), + } + + +def _late_transfer_target_bank_static_binding(*, mode: str) -> dict[str, object]: + """Return the shared non-recursive target-bank mode binding.""" + + if mode not in {"ephemeral_no_checkpoint", "identity_bound_checkpoint"}: + raise ValueError(f"Unsupported US late target-bank mode {mode!r}.") + return { + "resource_kind": "late_transfer_target_bank", + "schema_version": 1, + "mode": mode, + } + + +def _late_transfer_target_bank_binding( + *, + group_name: str, + target_bank: AcsTransferTargetBank | None, +) -> dict[str, object]: + """Return one runtime target-bank binding without hiding its mode.""" + + if target_bank is None: + return _late_transfer_target_bank_static_binding(mode="ephemeral_no_checkpoint") + identity_sha256 = getattr(target_bank, "identity_sha256", None) + _validate_sha256( + identity_sha256, + boundary=f"US late transfer {group_name!r} target-bank identity", + ) + return { + **_late_transfer_target_bank_static_binding(mode="identity_bound_checkpoint"), + "identity_sha256": identity_sha256, + } + + +def _late_transfer_resource_receipts( + *, + group_name: str, + entity: str, + family: str, + targets: Sequence[str], + seed: int, + n_estimators: int, + max_targets_per_fit: int, + target_bank: AcsTransferTargetBank | None, +) -> dict[str, dict[str, object]]: + """Bind model controls and durable-bank identity for one transfer node.""" + + model_binding = _late_transfer_model_config_binding( + group_name=group_name, + entity=entity, + family=family, + targets=targets, + seed=seed, + n_estimators=n_estimators, + max_targets_per_fit=max_targets_per_fit, + ) + bank_binding = _late_transfer_target_bank_binding( + group_name=group_name, + target_bank=target_bank, + ) + return { + f"{entity}.{US_LATE_TRANSFER_MODEL_CONFIG_INPUT}": ( + _late_available_input_receipt( + producer=group_name, + entity=entity, + column=US_LATE_TRANSFER_MODEL_CONFIG_INPUT, + rows=1, + binding=model_binding, + ) + ), + f"{entity}.{US_LATE_TRANSFER_TARGET_BANK_INPUT}": ( + _late_available_input_receipt( + producer=group_name, + entity=entity, + column=US_LATE_TRANSFER_TARGET_BANK_INPUT, + rows=1, + binding=bank_binding, + ) + ), + } + + +def stacked_late_producer_resource_semantics_receipt( + *, + clone_attachment_fraction: float, + clone_attachment_seed: int, + primary_seed: int, + primary_n_estimators: int, + transfer_seed: int, + transfer_n_estimators: int, + transfer_max_targets_per_fit: int, +) -> dict[str, object]: + """Bind every static or derivation-mode resource in the late DAG. + + Runtime table and bank digests remain dynamic. Their exact codecs and + derivations are bound here so the outer checkpoint identity covers the + complete resource doctrine without recursively embedding its own digest. + """ + + schedule_receipt = us_late_producer_schedule_receipt() + group_by_name = {group.name: group for group in CANONICAL_US_LATE_TRANSFER_GROUPS} + producer_rows: list[dict[str, object]] = [] + for producer_name in CANONICAL_US_LATE_PRODUCER_SCHEDULE.order: + contract = CANONICAL_US_LATE_PRODUCER_REGISTRY[producer_name] + expected_keys = _late_contract_available_input_keys(contract) + resources: dict[str, object] + if contract.kind == "acs_earnings_universe": + resources = { + f"person.{US_LATE_ACS_EARNINGS_UNIVERSE_CONFIG_INPUT}": { + "resolution": "static_exact", + "binding": _late_acs_earnings_universe_execution_binding(), + } + } + elif contract.kind == "primary_puf": + resources = { + "tax_unit.@puf_donor_tax_units": { + "resolution": "runtime_content_bound", + "binding": _late_puf_donor_resource_semantics_binding(), + }, + "tax_unit.@primary_qrf_checkpoint": { + "resolution": "outer_identity_derived", + "binding": _late_primary_qrf_checkpoint_static_binding(), + "dynamic_field": { + "name": "checkpoint_identity_sha256", + "derivation": ( + "canonical_sha256(outer_stacked_checkpoint_base_identity)" + ), + }, + }, + f"tax_unit.{US_LATE_PRIMARY_EXECUTION_CONFIG_INPUT}": { + "resolution": "static_exact", + "binding": _late_primary_execution_config_binding( + clone_attachment_fraction=clone_attachment_fraction, + clone_attachment_seed=clone_attachment_seed, + seed=primary_seed, + n_estimators=primary_n_estimators, + predictors=None, + person_outputs=None, + tax_unit_outputs=None, + fit_records_enabled=True, + tail_bound_diagnostics_enabled=True, + ), + }, + } + elif contract.kind == "post_clone_source": + resources = { + f"person.{US_LATE_SOURCE_EXECUTION_CONFIG_INPUT}": { + "resolution": "static_exact", + "binding": _late_source_execution_config_binding(producer_name), + } + } + elif contract.kind == "source_finalizer": + resources = { + f"person.{US_LATE_SOURCE_FINALIZER_EXECUTION_CONFIG_INPUT}": { + "resolution": "static_exact", + "binding": _late_source_finalizer_execution_binding(), + } + } + resources.update( + { + f"person.@source_receipt:{operator}": { + "resolution": "signed_callback_receipt_derived", + "binding": { + "resource_kind": "source_operator_receipt", + "schema_version": 1, + "source_operator": operator, + "dynamic_field": { + "name": "source_receipt_sha256", + "derivation": ( + "canonical_sha256(signed_source_callback_receipt)" + ), + }, + }, + } + for operator in POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + } + ) + elif contract.kind == "late_transfer": + group = group_by_name[producer_name] + resources = { + f"{group.entity}.{US_LATE_TRANSFER_MODEL_CONFIG_INPUT}": { + "resolution": "static_exact", + "binding": _late_transfer_model_config_binding( + group_name=group.name, + entity=group.entity, + family=group.family, + targets=group.targets, + seed=transfer_seed, + n_estimators=transfer_n_estimators, + max_targets_per_fit=transfer_max_targets_per_fit, + ), + }, + f"{group.entity}.{US_LATE_TRANSFER_TARGET_BANK_INPUT}": { + "resolution": "transferred_stage_identity_derived", + "binding": _late_transfer_target_bank_static_binding( + mode="identity_bound_checkpoint" + ), + "dynamic_field": { + "name": "identity_sha256", + "derivation": { + "base": "transferred_pool_checkpoint_stage_identity", + "late_producer_dag_sha256": schedule_receipt[ + "schedule_sha256" + ], + "late_producer_schedule_sha256": schedule_receipt[ + "payload_sha256" + ], + "producer": { + "name": group.name, + "entity": group.entity, + "family": group.family, + "ordered_targets": list(group.targets), + }, + }, + }, + }, + } + else: # pragma: no cover - import-validated canonical kind partition + raise AssertionError(f"Unhandled late producer kind {contract.kind!r}.") + if set(resources) != expected_keys: + raise RuntimeError( + f"US late resource semantics for {producer_name!r} do not cover " + f"its exact virtual surface; missing={sorted(expected_keys - set(resources))}, " + f"extra={sorted(set(resources) - expected_keys)}." + ) + producer_rows.append( + { + "producer": producer_name, + "kind": contract.kind, + "resources": { + key: _json_ready(resources[key]) for key in sorted(resources) + }, + } + ) + payload: dict[str, object] = { + "artifact_kind": _LATE_RESOURCE_SEMANTICS_ARTIFACT_KIND, + "schema_version": 1, + "producer_schedule_sha256": schedule_receipt["schedule_sha256"], + "producer_schedule_payload_sha256": schedule_receipt["payload_sha256"], + "producer_count": len(producer_rows), + "producers": producer_rows, + } + payload["sha256"] = _canonical_sha256(payload) + return payload + + +def _late_frame_content_sha256(frame: Frame) -> str: + """Hash a live frame while excluding the self-referential authority key.""" + + metadata = { + key: value + for key, value in frame.metadata.items() + if key != US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY + } + table_receipts = { + name: _late_table_values_sha256( + frame.table(name), + normalize_strings=True, + ) + for name in frame.entities + } + link_receipts = { + name: _late_table_values_sha256( + frame.link(name), + normalize_strings=True, + ) + for name in frame.links + } + weight_receipts = {} + for entity in frame.weighted_entities: + weights = frame.weights_for(entity) + digest = hashlib.sha256(weights.values.astype(" dict[str, object]: + """Bind one declared physical or virtual input to its live content.""" + + missing_rows, invalid_rows = _late_input_column_readiness_rows( + frame, + input_column=input_column, + required_scope=required_scope, + producer_name=producer_name, + available_input_receipts=available_input_receipts, + ) + payload: dict[str, object] = { + "entity": input_column.entity, + "column": input_column.column, + "value_kind": input_column.value_kind, + "missing_rows": missing_rows, + "invalid_rows": invalid_rows, + } + if input_column.entity == "frame": + metadata_key = input_column.column.removeprefix("@") + value = frame.metadata.get(metadata_key) + payload.update( + { + "required_scope": required_scope, + "scope_rows": 1, + "status": "present" if isinstance(value, Mapping) else "absent", + "content_sha256": _canonical_sha256( + _json_ready(value) + if isinstance(value, Mapping) + else {"absent": True} + ), + } + ) + return payload + table = frame.table(input_column.entity) + scope = _late_required_scope_mask( + frame, + entity=input_column.entity, + required_scope=required_scope, + ) + payload.update( + { + "required_scope": required_scope, + "scope_rows": int(scope.sum()), + } + ) + if input_column.column == "@resolved_weight": + weights = frame.resolve_weights(input_column.entity) + scoped = pd.DataFrame( + {"resolved_weight": weights.values[scope.to_numpy(dtype=bool)]}, + index=table.index[scope], + ) + payload.update( + { + "status": "present", + "weight_kind": weights.kind.value, + "content_sha256": _late_table_values_sha256(scoped), + } + ) + return payload + if input_column.column.startswith("@"): + receipt_key = f"{input_column.entity}.{input_column.column}" + receipt = available_input_receipts.get(receipt_key) + payload.update( + { + "status": "present" if isinstance(receipt, Mapping) else "absent", + "content_sha256": _canonical_sha256( + _json_ready(receipt) + if isinstance(receipt, Mapping) + else {"absent": True} + ), + } + ) + return payload + if input_column.column not in table: + payload.update( + { + "status": "absent", + "content_sha256": _canonical_sha256({"absent": True}), + } + ) + return payload + values = table.loc[scope, [input_column.column]] + payload.update( + { + "status": "present", + "content_sha256": _late_table_values_sha256(values), + } + ) + return payload + + +def _late_declared_input_evidence( + frame: Frame, + contract: ProducerContract, + *, + available_input_receipts: Mapping[str, Mapping[str, object]], + unfilled_rows: Mapping[ProducerInput, int], + invalid_rows: Mapping[ProducerInput, int], +) -> list[dict[str, object]]: + """Build the exact, content-bound readiness surface for one producer.""" + + result: list[dict[str, object]] = [] + for requirement in contract.inputs: + evidence = { + "alternatives": [ + [ + _late_input_column_evidence( + frame, + input_column=column, + required_scope=requirement.required_scope, + producer_name=contract.name, + available_input_receipts=available_input_receipts, + ) + for column in alternative + ] + for alternative in requirement.alternatives + ] + } + evidence["sha256"] = _canonical_sha256(evidence) + result.append( + { + "entity": requirement.entity, + "column": requirement.column, + "required_scope": requirement.required_scope, + "producing_stage": requirement.producing_stage, + "unfilled_rows": unfilled_rows[requirement], + "invalid_rows": invalid_rows[requirement], + "evidence": evidence, + } + ) + return result + + +def _late_scalar_is_hashable(value: object) -> bool: + try: + hash(value) + except TypeError: + return False + return True + + +def _late_output_matches_metric_family(values: pd.Series, metric: str) -> bool: + """Return whether one late output's physical dtype matches its registry family.""" + + is_boolean = pd.api.types.is_bool_dtype(values.dtype) + is_real_numeric = bool( + pd.api.types.is_numeric_dtype(values.dtype) + and not is_boolean + and not pd.api.types.is_complex_dtype(values.dtype) + ) + if metric == "boolean_incidence": + return bool(is_boolean) + if metric in {"monetary_sign_separated", "rare_incidence"}: + return is_real_numeric + if metric != "categorical_tvd": + return False + if is_boolean or pd.api.types.is_complex_dtype(values.dtype): + return False + if is_real_numeric or isinstance(values.dtype, pd.CategoricalDtype): + return True + if pd.api.types.is_object_dtype(values.dtype): + inferred = pd.api.types.infer_dtype(values, skipna=True) + return inferred != "boolean" and all( + _late_scalar_is_hashable(value) for value in values.dropna() + ) + return bool(pd.api.types.is_string_dtype(values.dtype)) + + +def _validate_late_callback_output_metric_families( + frame: Frame, + contract: ProducerContract, + *, + metric_registry: Mapping[ + tuple[str, str, str, int], str + ] = _CANONICAL_ORIGIN_BATTERY_METRIC_REGISTRY_ANCHOR, +) -> None: + """Fail closed when a callback materializes a registered output incorrectly.""" + + declared_by_column: dict[tuple[str, str], list[tuple[str, str]]] = {} + for (entity, family, column, clone_index), metric in metric_registry.items(): + if clone_index != 0: + continue + declared_by_column.setdefault((entity, column), []).append((family, metric)) + + failures: list[str] = [] + for output in contract.outputs: + if output.entity == "frame" or output.column.startswith("@"): + continue + declarations = declared_by_column.get((output.entity, output.column), ()) + if not declarations: + continue + metrics = {metric for _family, metric in declarations} + if len(metrics) != 1: + raise RuntimeError( + "The origin-battery metric registry gives conflicting dtype " + f"families to {output.entity}.{output.column}: {declarations}." + ) + table = frame.table(output.entity) + if output.column not in table: + continue + metric = next(iter(metrics)) + values = table[output.column] + if not _late_output_matches_metric_family(values, metric): + families = sorted(family for family, _metric in declarations) + failures.append( + f"{output.entity}.{output.column}: registry families {families} " + f"declare {metric!r}, callback dtype is {str(values.dtype)!r}" + ) + if failures: + raise TypeError( + f"Late producer {contract.name!r} output dtype-family validation " + "failed:\n " + "\n ".join(failures) + ) + + +def _late_output_column_evidence( + frame: Frame, + *, + output: ProducerOutput, + producer_receipt: Mapping[str, object], +) -> dict[str, object]: + """Bind one declared producer output to its post-callback content.""" + + payload: dict[str, object] = { + "entity": output.entity, + "column": output.column, + "coverage_scope": output.coverage_scope, + } + if output.entity == "frame": + value = frame.metadata.get(output.column.removeprefix("@")) + payload.update( + { + "status": "present" if isinstance(value, Mapping) else "absent", + "content_sha256": _canonical_sha256( + _json_ready(value) + if isinstance(value, Mapping) + else {"absent": True} + ), + } + ) + return payload + table = frame.table(output.entity) + if output.column.startswith("@source_receipt:"): + payload.update( + { + "scope_rows": len(table), + "status": "present", + "content_sha256": _canonical_sha256(_json_ready(producer_receipt)), + } + ) + return payload + scope = _late_required_scope_mask( + frame, + entity=output.entity, + required_scope=output.coverage_scope, + ) + payload["scope_rows"] = int(scope.sum()) + if output.column == "@resolved_weight": + weights = frame.resolve_weights(output.entity) + scoped = pd.DataFrame( + {"resolved_weight": weights.values[scope.to_numpy(dtype=bool)]}, + index=table.index[scope], + ) + payload.update( + { + "status": "present", + "weight_kind": weights.kind.value, + "content_sha256": _late_table_values_sha256(scoped), + } + ) + elif output.column not in table: + payload.update( + { + "status": "absent", + "content_sha256": _canonical_sha256({"absent": True}), + } + ) + else: + payload.update( + { + "status": "present", + "content_sha256": _late_table_values_sha256( + table.loc[scope, [output.column]] + ), + } + ) + return payload + + +def _late_producer_transition_authority_receipt( + receipt: Mapping[str, object], +) -> dict[str, object]: + """Derive the immutable live-frame anchor for one signed DAG receipt.""" + + schedule = receipt["producer_schedule"] + assert isinstance(schedule, Mapping) + authority: dict[str, object] = { + "authority_id": US_LATE_PRODUCER_TRANSITION_AUTHORITY_ID, + "version": US_LATE_PRODUCER_TRANSITION_AUTHORITY_VERSION, + "receipt_sha256": receipt["sha256"], + "producer_schedule_sha256": schedule["payload_sha256"], + "input_frame_sha256": receipt["input_frame_sha256"], + "output_frame_sha256": receipt["output_frame_sha256"], + "execution_chain_sha256": receipt["execution_chain_sha256"], + } + authority["sha256"] = _canonical_sha256(authority) + return authority + + +def _bind_late_producer_transition_authority( + frame: Frame, + receipt: Mapping[str, object], +) -> tuple[Frame, str]: + """Bind a generated DAG transition into immutable Frame metadata.""" + + if US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY in frame.metadata: + raise ValueError( + "US late-producer transition authority is already bound; refusing " + "to overwrite the immutable generation anchor." + ) + authority = _late_producer_transition_authority_receipt(receipt) + tables = {entity: frame.table(entity).copy() for entity in frame.entities} + tables.update({link: frame.link(link).copy() for link in frame.links}) + bound = Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata={ + **frame.metadata, + US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY: authority, + }, + ) + return bound, str(authority["sha256"]) + + +def _validate_primary_callback_resource_binding( + producer_receipt: Mapping[str, object], + *, + available_input_receipts: Mapping[str, object], + boundary: str, +) -> None: + """Prove the primary callback consumed the resources gated by its DAG row.""" + + observed = producer_receipt.get("primary_resource_receipts_sha256") + _validate_sha256(observed, boundary=f"{boundary} primary callback resources") + expected = _canonical_sha256(_json_ready(available_input_receipts)) + if observed != expected: + raise ValueError( + f"{boundary}: primary callback receipt disagrees with its declared " + f"resources; expected={expected}, observed={observed}." + ) + + +def _validate_late_execution_row( + raw_row: object, + *, + contract: ProducerContract, + execution_index: int, + expected_previous_sha256: str, + boundary: str, +) -> str: + """Re-run one persisted readiness proof without invoking its callback.""" + + if not isinstance(raw_row, Mapping): + raise ValueError( + f"{boundary}: late producer execution row {execution_index} is not " + "an object." + ) + expected_keys = { + "execution_index", + "producer", + "kind", + "declared_inputs", + "declared_absence_receipts", + "available_input_receipts", + "input_surface_sha256", + "output_surface", + "output_surface_sha256", + "producer_receipt", + "producer_receipt_sha256", + "previous_execution_sha256", + "status", + "sha256", + } + if set(raw_row) != expected_keys: + raise ValueError( + f"{boundary}: late producer execution row {execution_index} schema " + f"drifted; missing={sorted(expected_keys - set(raw_row))}, " + f"extra={sorted(set(raw_row) - expected_keys)}." + ) + expected_status = "complete" + if ( + raw_row.get("execution_index") != execution_index + or raw_row.get("producer") != contract.name + or raw_row.get("kind") != contract.kind + or raw_row.get("status") != expected_status + ): + raise ValueError( + f"{boundary}: late producer execution row {execution_index} is " + f"misbound; expected producer={contract.name!r}, kind={contract.kind!r}, " + f"status={expected_status!r}." + ) + declared_inputs = raw_row.get("declared_inputs") + if not isinstance(declared_inputs, list) or len(declared_inputs) != len( + contract.inputs + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} does not carry its " + f"exact {len(contract.inputs)}-input readiness surface." + ) + unfilled_rows: dict[ProducerInput, int] = {} + invalid_rows: dict[ProducerInput, int] = {} + evidenced_available_keys: set[str] = set() + evidenced_available_sha256: dict[str, str] = {} + physical_input_states: dict[ + tuple[str, str, str], tuple[int, int, str, str, str | None] + ] = {} + scope_cardinalities: dict[tuple[str, str], int] = {} + for requirement, raw_input in zip(contract.inputs, declared_inputs, strict=True): + if not isinstance(raw_input, Mapping): + raise ValueError( + f"{boundary}: late producer {contract.name!r} has a malformed " + "declared-input receipt." + ) + expected_input = { + "entity": requirement.entity, + "column": requirement.column, + "required_scope": requirement.required_scope, + "producing_stage": requirement.producing_stage, + } + expected_input_keys = { + *expected_input, + "unfilled_rows", + "invalid_rows", + "evidence", + } + if set(raw_input) != expected_input_keys: + raise ValueError( + f"{boundary}: late producer {contract.name!r} input receipt " + f"schema drifted from {requirement.entity}.{requirement.column}." + ) + if any(raw_input.get(key) != value for key, value in expected_input.items()): + raise ValueError( + f"{boundary}: late producer {contract.name!r} input receipt " + f"drifted from {requirement.entity}.{requirement.column}." + ) + rows = raw_input.get("unfilled_rows") + if isinstance(rows, bool) or not isinstance(rows, int) or rows < 0: + raise ValueError( + f"{boundary}: late producer {contract.name!r} input " + f"{requirement.entity}.{requirement.column} has invalid " + f"unfilled_rows={rows!r}." + ) + unfilled_rows[requirement] = rows + invalid = raw_input.get("invalid_rows") + if isinstance(invalid, bool) or not isinstance(invalid, int) or invalid < 0: + raise ValueError( + f"{boundary}: late producer {contract.name!r} input " + f"{requirement.entity}.{requirement.column} has invalid " + f"invalid_rows={invalid!r}." + ) + invalid_rows[requirement] = invalid + evidence = raw_input.get("evidence") + if not isinstance(evidence, Mapping) or set(evidence) != { + "alternatives", + "sha256", + }: + raise ValueError( + f"{boundary}: late producer {contract.name!r} input " + f"{requirement.entity}.{requirement.column} has malformed " + "content evidence." + ) + evidence_unsigned = dict(evidence) + evidence_sha256 = evidence_unsigned.pop("sha256") + _validate_sha256( + evidence_sha256, + boundary=(f"{boundary} late producer {contract.name!r} input evidence"), + ) + if evidence_sha256 != _canonical_sha256(evidence_unsigned): + raise ValueError( + f"{boundary}: late producer {contract.name!r} input " + f"{requirement.entity}.{requirement.column} evidence SHA-256 " + "mismatch." + ) + alternatives = evidence.get("alternatives") + if not isinstance(alternatives, list) or len(alternatives) != len( + requirement.alternatives + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} input " + f"{requirement.entity}.{requirement.column} evidence changed " + "its alternative surface." + ) + column_states: dict[ + ProducerInputColumn, tuple[int, int, dict[str, object]] + ] = {} + alternative_missing_counts: list[int] = [] + for declared_alternative, raw_alternative in zip( + requirement.alternatives, + alternatives, + strict=True, + ): + if not isinstance(raw_alternative, list) or len(raw_alternative) != len( + declared_alternative + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} input " + f"{requirement.entity}.{requirement.column} evidence changed " + "one physical alternative." + ) + alternative_missing = 0 + for declared_column, raw_column in zip( + declared_alternative, + raw_alternative, + strict=True, + ): + if not isinstance(raw_column, Mapping) or any( + raw_column.get(field) != value + for field, value in { + "entity": declared_column.entity, + "column": declared_column.column, + "value_kind": declared_column.value_kind, + "required_scope": requirement.required_scope, + }.items() + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} input " + f"{requirement.entity}.{requirement.column} evidence " + "is misbound to its physical columns." + ) + expected_column_keys = { + "entity", + "column", + "value_kind", + "required_scope", + "scope_rows", + "missing_rows", + "invalid_rows", + "status", + "content_sha256", + } + if declared_column.column == "@resolved_weight": + expected_column_keys.add("weight_kind") + if set(raw_column) != expected_column_keys: + raise ValueError( + f"{boundary}: late producer {contract.name!r} input " + f"{requirement.entity}.{requirement.column} evidence " + f"schema drifted for {declared_column.entity}." + f"{declared_column.column}." + ) + for count_field in ("scope_rows", "missing_rows", "invalid_rows"): + count = raw_column.get(count_field) + if ( + isinstance(count, bool) + or not isinstance(count, int) + or count < 0 + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} " + f"input evidence has invalid {count_field}={count!r}." + ) + scope_rows = int(raw_column["scope_rows"]) + missing_rows = int(raw_column["missing_rows"]) + evidence_invalid_rows = int(raw_column["invalid_rows"]) + status = raw_column.get("status") + if status not in {"present", "absent"}: + raise ValueError( + f"{boundary}: late producer {contract.name!r} input " + f"evidence has invalid status={status!r}." + ) + is_virtual = ( + declared_column.column.startswith("@") + and declared_column.column != "@resolved_weight" + and declared_column.entity != "frame" + ) + if declared_column.entity == "frame" and scope_rows != 1: + raise ValueError( + f"{boundary}: late producer {contract.name!r} frame " + "input evidence must have scope_rows=1." + ) + if status == "absent": + expected_missing = ( + 1 + if declared_column.entity == "frame" + else max(1, scope_rows) + if is_virtual + else scope_rows + ) + if ( + missing_rows != expected_missing + or evidence_invalid_rows != 0 + or raw_column.get("content_sha256") + != _canonical_sha256({"absent": True}) + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} " + "absent input evidence has inconsistent counts or " + "content identity." + ) + elif declared_column.entity == "frame" or is_virtual: + if missing_rows != 0 or evidence_invalid_rows != 0: + raise ValueError( + f"{boundary}: late producer {contract.name!r} " + "present virtual input evidence has nonzero " + "readiness counts." + ) + elif declared_column.column == "@resolved_weight": + if ( + missing_rows != 0 + or not isinstance(raw_column.get("weight_kind"), str) + or not raw_column.get("weight_kind") + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} " + "resolved-weight evidence is malformed." + ) + elif ( + missing_rows > scope_rows + or evidence_invalid_rows > scope_rows + or missing_rows + evidence_invalid_rows > scope_rows + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} physical " + "input evidence counts exceed its declared scope." + ) + _validate_sha256( + raw_column.get("content_sha256"), + boundary=( + f"{boundary} late producer {contract.name!r} input content" + ), + ) + normalized_column = dict(raw_column) + if ( + declared_column.entity != "frame" + and not declared_column.column.startswith("@") + ): + physical_key = ( + declared_column.entity, + declared_column.column, + requirement.required_scope, + ) + physical_state = ( + scope_rows, + missing_rows, + str(status), + str(raw_column["content_sha256"]), + str(raw_column["weight_kind"]) + if "weight_kind" in raw_column + else None, + ) + previous_physical_state = physical_input_states.get(physical_key) + if ( + previous_physical_state is not None + and previous_physical_state != physical_state + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} carries " + "inconsistent physical input evidence for " + f"{declared_column.entity}.{declared_column.column} " + f"on {requirement.required_scope}." + ) + physical_input_states[physical_key] = physical_state + scope_key = ( + declared_column.entity, + requirement.required_scope, + ) + previous_scope_rows = scope_cardinalities.get(scope_key) + if ( + previous_scope_rows is not None + and previous_scope_rows != scope_rows + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} has " + "inconsistent input scope cardinality for " + f"{declared_column.entity}.{requirement.required_scope}." + ) + scope_cardinalities[scope_key] = scope_rows + previous_state = column_states.get(declared_column) + current_state = ( + missing_rows, + evidence_invalid_rows, + normalized_column, + ) + if previous_state is not None and previous_state != current_state: + raise ValueError( + f"{boundary}: late producer {contract.name!r} repeats " + "one physical input with inconsistent evidence." + ) + column_states[declared_column] = current_state + alternative_missing += missing_rows + if ( + declared_column.column.startswith("@") + and declared_column.column != "@resolved_weight" + and declared_column.entity != "frame" + and raw_column.get("status") == "present" + ): + evidence_key = f"{declared_column.entity}.{declared_column.column}" + evidenced_available_keys.add(evidence_key) + evidenced_available_sha256[evidence_key] = str( + raw_column["content_sha256"] + ) + alternative_missing_counts.append(alternative_missing) + recomputed_unfilled = min(alternative_missing_counts) + recomputed_invalid = sum(state[1] for state in column_states.values()) + if rows != recomputed_unfilled or invalid != recomputed_invalid: + raise ValueError( + f"{boundary}: late producer {contract.name!r} input " + f"{requirement.entity}.{requirement.column} readiness counts " + "disagree with its physical evidence; " + f"declared=({rows}, {invalid}), " + f"recomputed=({recomputed_unfilled}, {recomputed_invalid})." + ) + + raw_absence = raw_row.get("declared_absence_receipts") + if not isinstance(raw_absence, Mapping): + raise ValueError( + f"{boundary}: late producer {contract.name!r} absence receipts are " + "not an object." + ) + expected_absence_ids = { + receipt_id + for requirement, rows in unfilled_rows.items() + if rows > 0 and invalid_rows[requirement] == 0 + for receipt_id in requirement.tolerated_absence_receipts + } + if set(raw_absence) != expected_absence_ids: + raise ValueError( + f"{boundary}: late producer {contract.name!r} declared-absence " + f"surface drifted; expected={sorted(expected_absence_ids)}, " + f"got={sorted(map(str, raw_absence))}." + ) + for requirement, rows in unfilled_rows.items(): + if rows <= 0: + continue + for receipt_id in requirement.tolerated_absence_receipts: + receipt = raw_absence.get(receipt_id) + expected_receipt = { + "receipt_id": receipt_id, + "status": "declared_absence", + "entity": requirement.entity, + "column": requirement.column, + "required_scope": requirement.required_scope, + "rows": rows, + "producer": contract.name, + "reason": "optional availability-pattern input", + } + if not isinstance(receipt, Mapping) or dict(receipt) != expected_receipt: + raise ValueError( + f"{boundary}: late producer {contract.name!r} absence receipt " + f"{receipt_id!r} is not canonical." + ) + + available_inputs = raw_row.get("available_input_receipts") + if not isinstance(available_inputs, Mapping): + raise ValueError( + f"{boundary}: late producer {contract.name!r} available-input " + "receipts are not an object." + ) + if contract.kind in { + "acs_earnings_universe", + "primary_puf", + "source_finalizer", + "late_transfer", + }: + mandatory_available_keys = _late_contract_available_input_keys(contract) + elif contract.kind == "post_clone_source": + mandatory_available_keys = {f"person.{US_LATE_SOURCE_EXECUTION_CONFIG_INPUT}"} + else: + mandatory_available_keys = set() + if not mandatory_available_keys <= evidenced_available_keys: + raise ValueError( + f"{boundary}: late producer {contract.name!r} virtual-input " + "evidence does not prove every mandatory available resource." + ) + if contract.kind in { + "acs_earnings_universe", + "primary_puf", + "source_finalizer", + "late_transfer", + }: + if evidenced_available_keys != mandatory_available_keys: + raise ValueError( + f"{boundary}: late producer {contract.name!r} virtual-input " + "evidence adds a noncanonical mandatory resource." + ) + expected_available_keys = evidenced_available_keys + if set(available_inputs) != expected_available_keys: + raise ValueError( + f"{boundary}: late producer {contract.name!r} available-input " + f"surface drifted; expected={sorted(expected_available_keys)}, " + f"got={sorted(map(str, available_inputs))}." + ) + for key, receipt in available_inputs.items(): + entity, column = key.split(".", 1) + _validate_late_available_input_receipt( + receipt, + producer=contract.name, + entity=entity, + column=column, + boundary=f"{boundary} late producer {contract.name!r}", + ) + if evidenced_available_sha256.get(key) != _canonical_sha256( + _json_ready(receipt) + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} available-input " + f"receipt {key!r} disagrees with its declared content evidence." + ) + + input_surface_sha256 = raw_row.get("input_surface_sha256") + _validate_sha256( + input_surface_sha256, + boundary=f"{boundary} late producer {contract.name!r} input surface", + ) + if input_surface_sha256 != _canonical_sha256(declared_inputs): + raise ValueError( + f"{boundary}: late producer {contract.name!r} input-surface " + "SHA-256 mismatch." + ) + + output_surface = raw_row.get("output_surface") + if not isinstance(output_surface, list) or len(output_surface) != len( + contract.outputs + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} does not carry its " + f"exact {len(contract.outputs)}-output content surface." + ) + for output, raw_output in zip(contract.outputs, output_surface, strict=True): + expected_output = { + "entity": output.entity, + "column": output.column, + "coverage_scope": output.coverage_scope, + } + expected_output_keys = { + *expected_output, + "status", + "content_sha256", + } + if output.entity != "frame": + expected_output_keys.add("scope_rows") + if output.column == "@resolved_weight": + expected_output_keys.add("weight_kind") + if ( + not isinstance(raw_output, Mapping) + or set(raw_output) != expected_output_keys + or any( + raw_output.get(field) != value + for field, value in expected_output.items() + ) + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} output evidence " + f"drifted from {output.entity}.{output.column}." + ) + if raw_output.get("status") != "present": + raise ValueError( + f"{boundary}: late producer {contract.name!r} completed with " + f"absent output {output.entity}.{output.column}." + ) + if output.entity != "frame": + scope_rows = raw_output.get("scope_rows") + if ( + isinstance(scope_rows, bool) + or not isinstance(scope_rows, int) + or scope_rows < 0 + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} output " + f"{output.entity}.{output.column} has invalid " + f"scope_rows={scope_rows!r}." + ) + if contract.kind != "primary_puf" and not output.column.startswith("@"): + scope_key = (output.entity, output.coverage_scope) + previous_scope_rows = scope_cardinalities.get(scope_key) + if ( + previous_scope_rows is not None + and previous_scope_rows != scope_rows + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} output " + "scope cardinality disagrees with its declared input " + f"scope for {output.entity}.{output.coverage_scope}; " + f"input={previous_scope_rows}, output={scope_rows}." + ) + scope_cardinalities[scope_key] = int(scope_rows) + if output.column == "@resolved_weight" and ( + not isinstance(raw_output.get("weight_kind"), str) + or not raw_output.get("weight_kind") + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} resolved-weight " + "output evidence is malformed." + ) + _validate_sha256( + raw_output.get("content_sha256"), + boundary=f"{boundary} late producer {contract.name!r} output content", + ) + output_surface_sha256 = raw_row.get("output_surface_sha256") + _validate_sha256( + output_surface_sha256, + boundary=f"{boundary} late producer {contract.name!r} output surface", + ) + if output_surface_sha256 != _canonical_sha256(output_surface): + raise ValueError( + f"{boundary}: late producer {contract.name!r} output-surface " + "SHA-256 mismatch." + ) + + producer_receipt = raw_row.get("producer_receipt") + if not isinstance(producer_receipt, Mapping): + raise ValueError( + f"{boundary}: late producer {contract.name!r} callback receipt is " + "not an object." + ) + producer_receipt_sha256 = raw_row.get("producer_receipt_sha256") + _validate_sha256( + producer_receipt_sha256, + boundary=f"{boundary} late producer {contract.name!r} callback receipt", + ) + if producer_receipt_sha256 != _canonical_sha256(producer_receipt): + raise ValueError( + f"{boundary}: late producer {contract.name!r} callback-receipt " + "SHA-256 mismatch." + ) + for raw_output in output_surface: + if str(raw_output["column"]).startswith("@source_receipt:") and ( + raw_output["content_sha256"] != producer_receipt_sha256 + ): + raise ValueError( + f"{boundary}: late producer {contract.name!r} source-receipt " + "output digest disagrees with its callback receipt." + ) + if contract.kind == "primary_puf": + _validate_primary_callback_resource_binding( + producer_receipt, + available_input_receipts=available_inputs, + boundary=f"{boundary} late producer {contract.name!r}", + ) + + if raw_row.get("previous_execution_sha256") != expected_previous_sha256: + raise ValueError( + f"{boundary}: late producer {contract.name!r} execution chain does " + "not name the preceding digest." + ) + + run_producer_when_ready( + contract, + lambda: None, + unfilled_rows=unfilled_rows, + invalid_rows=invalid_rows, + absence_receipts=raw_absence, + ) + unsigned = dict(raw_row) + observed_sha256 = unsigned.pop("sha256") + _validate_sha256( + observed_sha256, + boundary=f"{boundary} late producer {contract.name!r} execution row", + ) + if observed_sha256 != _canonical_sha256(unsigned): + raise ValueError( + f"{boundary}: late producer {contract.name!r} execution-row " + "SHA-256 mismatch." + ) + return str(observed_sha256) + + +def _late_execution_genesis_sha256( + *, + producer_schedule_sha256: object, + input_frame_sha256: object, +) -> str: + return _canonical_sha256( + { + "receipt_schema_version": US_LATE_PRODUCER_RECEIPT_SCHEMA_VERSION, + "producer_schedule_sha256": producer_schedule_sha256, + "input_frame_sha256": input_frame_sha256, + } + ) + + +def _validate_late_transition_authority( + frame: Frame, + receipt: Mapping[str, object], + *, + boundary: str, + expected_transition_authority_sha256: str, + require_live_output: bool, +) -> None: + """Authenticate the immutable frame anchor and independent digest carrier.""" + + _validate_sha256( + expected_transition_authority_sha256, + boundary=f"{boundary} independently carried late-producer authority", + ) + authority = frame.metadata.get(US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY) + if not isinstance(authority, Mapping): + raise ValueError( + f"{boundary}: late-producer transition authority is absent from " + "live frame metadata." + ) + expected_authority = _late_producer_transition_authority_receipt(receipt) + if dict(authority) != expected_authority: + raise ValueError( + f"{boundary}: late-producer receipt is not bound to the immutable " + "live transition authority." + ) + if authority.get("sha256") != expected_transition_authority_sha256: + raise ValueError( + f"{boundary}: late-producer receipt differs from the independently " + "carried late-producer transition authority." + ) + if require_live_output: + live_output_sha256 = _late_frame_content_sha256(frame) + if receipt.get("output_frame_sha256") != live_output_sha256: + raise ValueError( + f"{boundary}: late-producer output digest does not match the " + "live frame." + ) + + +def validate_stacked_late_producer_transition_authority( + frame: Frame, + receipt: Mapping[str, object], + *, + boundary: str, + expected_transition_authority_sha256: str, +) -> None: + """Validate the anchor after declared downstream operators have run.""" + + validate_stacked_late_producer_receipt(receipt, boundary=boundary) + _validate_late_transition_authority( + frame, + receipt, + boundary=boundary, + expected_transition_authority_sha256=(expected_transition_authority_sha256), + require_live_output=False, + ) + + +def validate_stacked_late_producer_receipt( + receipt: Mapping[str, object], + *, + boundary: str, + frame: Frame | None = None, + expected_transition_authority_sha256: str | None = None, +) -> None: + """Authenticate the complete derived execution and source/transfer proof.""" + + if not isinstance(receipt, Mapping): + raise ValueError(f"{boundary}: stacked late-producer DAG receipt is absent.") + expected_keys = { + "version", + "producer_schedule", + "input_frame_sha256", + "output_frame_sha256", + "execution_chain_sha256", + "execution", + "source_completion", + "post_puf_transfer", + "sha256", + } + if set(receipt) != expected_keys: + raise ValueError( + f"{boundary}: stacked late-producer DAG receipt schema drifted; " + f"missing={sorted(expected_keys - set(receipt))}, " + f"extra={sorted(set(receipt) - expected_keys)}." + ) + if receipt.get("version") != US_LATE_PRODUCER_RECEIPT_SCHEMA_VERSION: + raise ValueError( + f"{boundary}: stacked late-producer DAG receipt version changed." + ) + expected_schedule = _json_ready(us_late_producer_schedule_receipt()) + schedule = receipt.get("producer_schedule") + if not isinstance(schedule, Mapping) or _json_ready(schedule) != expected_schedule: + raise ValueError( + f"{boundary}: stacked late-producer DAG schedule is not canonical." + ) + for digest_field in ( + "input_frame_sha256", + "output_frame_sha256", + "execution_chain_sha256", + "sha256", + ): + _validate_sha256( + receipt.get(digest_field), boundary=f"{boundary} {digest_field}" + ) + execution = receipt.get("execution") + expected_order = CANONICAL_US_LATE_PRODUCER_SCHEDULE.order + if not isinstance(execution, list) or len(execution) != len(expected_order): + raise ValueError( + f"{boundary}: stacked late-producer DAG must carry exactly " + f"{len(expected_order)} execution rows." + ) + previous_sha256 = _late_execution_genesis_sha256( + producer_schedule_sha256=schedule["payload_sha256"], + input_frame_sha256=receipt["input_frame_sha256"], + ) + execution_by_name: dict[str, Mapping[str, object]] = {} + for index, producer_name in enumerate(expected_order): + raw_row = execution[index] + previous_sha256 = _validate_late_execution_row( + raw_row, + contract=CANONICAL_US_LATE_PRODUCER_REGISTRY[producer_name], + execution_index=index, + expected_previous_sha256=previous_sha256, + boundary=boundary, + ) + assert isinstance(raw_row, Mapping) + execution_by_name[producer_name] = raw_row + if receipt.get("execution_chain_sha256") != previous_sha256: + raise ValueError( + f"{boundary}: stacked late-producer execution-chain terminus " + "does not match its final row." + ) + + expected_source_order = [ + producer.removeprefix("source:") + for producer in expected_order + if producer.startswith("source:") + ] + source_completion = receipt.get("source_completion") + if not isinstance(source_completion, Mapping): + raise ValueError( + f"{boundary}: stacked late-producer DAG source completion is absent." + ) + suboperators = source_completion.get("suboperators") + if ( + source_completion.get("phase") != "post_clone" + or source_completion.get("operator_order") != expected_source_order + or not isinstance(suboperators, list) + or len(suboperators) != len(expected_source_order) + ): + raise ValueError( + f"{boundary}: stacked late-producer DAG source completion does not " + "match the derived sixteen-source order." + ) + for index, (operator, suboperator) in enumerate( + zip(expected_source_order, suboperators, strict=True) + ): + if ( + not isinstance(suboperator, Mapping) + or suboperator.get("operator") != operator + or suboperator.get("order_index") != index + ): + raise ValueError( + f"{boundary}: stacked source completion row {index} is not " + f"bound to {operator!r}." + ) + deferred = source_completion.get("deferred_transfer_inputs") + deferred_inputs = deferred.get("inputs") if isinstance(deferred, Mapping) else None + if not isinstance(deferred_inputs, Mapping) or set(deferred_inputs) != { + "bank_account_assets", + "bond_assets", + "stock_assets", + }: + raise ValueError( + f"{boundary}: stacked source completion lacks the exact three-input " + "deferred-source receipt." + ) + + normalized_suboperators: list[dict[str, object]] = [] + source_evidence: list[object] = [] + for index, operator in enumerate(expected_source_order): + source_row = execution_by_name[f"source:{operator}"] + source_receipt = source_row.get("producer_receipt") + if not isinstance(source_receipt, Mapping): + raise ValueError( + f"{boundary}: source producer {operator!r} has no callback receipt." + ) + source_suboperators = source_receipt.get("suboperators") + if ( + source_receipt.get("phase") != "post_clone" + or source_receipt.get("operator_order") != [operator] + or not isinstance(source_suboperators, list) + or len(source_suboperators) != 1 + or not isinstance(source_suboperators[0], Mapping) + or source_suboperators[0].get("operator") != operator + ): + raise ValueError( + f"{boundary}: source producer {operator!r} callback receipt is " + "not the canonical single-operator proof." + ) + normalized = dict(source_suboperators[0]) + normalized["order_index"] = index + normalized_suboperators.append(normalized) + source_evidence.append(source_receipt.get("cps_source_evidence")) + if suboperators != normalized_suboperators: + raise ValueError( + f"{boundary}: stacked source completion is not reconstructed from " + "the sixteen producer receipts." + ) + if source_evidence and any( + evidence != source_evidence[0] for evidence in source_evidence[1:] + ): + raise ValueError( + f"{boundary}: stacked source producer receipts disagree on CPS " + "source evidence." + ) + if source_completion.get("cps_source_evidence") != ( + source_evidence[0] if source_evidence else None + ): + raise ValueError( + f"{boundary}: stacked source completion CPS evidence is not bound " + "to its producer receipts." + ) + finalizer_row = execution_by_name[US_LATE_SOURCE_FINALIZER_STAGE] + finalizer_inputs = finalizer_row.get("available_input_receipts") + assert isinstance(finalizer_inputs, Mapping) + for operator in expected_source_order: + key = f"person.@source_receipt:{operator}" + source_receipt = execution_by_name[f"source:{operator}"]["producer_receipt"] + input_receipt = finalizer_inputs.get(key) + input_binding = ( + input_receipt.get("binding") if isinstance(input_receipt, Mapping) else None + ) + if not isinstance(input_binding, Mapping) or input_binding.get( + "source_receipt_sha256" + ) != _canonical_sha256(source_receipt): + raise ValueError( + f"{boundary}: source finalizer input {key!r} is not bound to " + "its producer receipt." + ) + if _json_ready(finalizer_row["producer_receipt"]) != _json_ready(source_completion): + raise ValueError( + f"{boundary}: source-finalizer execution receipt differs from the " + "aggregate source completion proof." + ) + + transfer = receipt.get("post_puf_transfer") + if not isinstance(transfer, Mapping): + raise ValueError( + f"{boundary}: stacked late-producer DAG transfer proof is absent." + ) + validate_stacked_post_puf_transfer_receipt(transfer, boundary=boundary) + groups = transfer["groups"] + assert isinstance(groups, Mapping) + canonical_family = { + (entity, target): family + for entity, families in CANONICAL_STACKED_POST_PUF_TRANSFER_SURFACE.items() + for family, targets in families.items() + for target in targets + } + reconstructed_targets: dict[str, object] = {} + for group in CANONICAL_US_LATE_TRANSFER_GROUPS: + row_receipt = execution_by_name[group.name]["producer_receipt"] + if _json_ready(row_receipt) != _json_ready(groups[group.name]): + raise ValueError( + f"{boundary}: transfer execution receipt for {group.name!r} " + "differs from its aggregate group proof." + ) + assert isinstance(row_receipt, Mapping) + group_targets = row_receipt.get("targets") + assert isinstance(group_targets, Mapping) + for target in group.targets: + bounded_label = f"{group.entity}/{group.family}/{target}" + aggregate_label = ( + f"{group.entity}/{canonical_family[(group.entity, target)]}/{target}" + ) + reconstructed_targets[aggregate_label] = group_targets[bounded_label] + if _json_ready(transfer["targets"]) != _json_ready(reconstructed_targets): + raise ValueError( + f"{boundary}: aggregate late-transfer targets are not reconstructed " + "from the nineteen producer receipts." + ) + + unsigned = dict(receipt) + observed_sha256 = unsigned.pop("sha256") + if observed_sha256 != _canonical_sha256(unsigned): + raise ValueError( + f"{boundary}: stacked late-producer DAG receipt SHA-256 mismatch." + ) + if (frame is None) != (expected_transition_authority_sha256 is None): + raise ValueError( + f"{boundary}: live frame and independently carried late-producer " + "authority must be supplied together." + ) + if frame is not None: + assert expected_transition_authority_sha256 is not None + _validate_late_transition_authority( + frame, + receipt, + boundary=boundary, + expected_transition_authority_sha256=(expected_transition_authority_sha256), + require_live_output=True, + ) + + +def _validate_test_authority(authority: _StackedAuthority, *, boundary: str) -> None: + """Keep the explicit fixture seam visibly and terminally non-production.""" + + receipt = _authority_receipt(authority) + if ( + authority.declared_form != _NONCANONICAL_AUTHORITY_FORM + or receipt["authority_form"] != _NONCANONICAL_AUTHORITY_FORM + or receipt["canonical_identity"] is not False + ): + raise ValueError(f"{boundary} requires a NON-CANONICAL test authority.") + + +def _validate_stacked_gate_manifest_details( + gate_name: str, + details: Mapping[str, object], + *, + passed: bool, + _canonical_surface: TargetFamilies = CANONICAL_STACKED_DECLARED_SURFACE, + _canonical_plan: tuple[GapFillDirection, ...] = CANONICAL_STACKED_GAP_FILL_PLAN, + _canonical_registry: Mapping[ + tuple[str, str, str, int], str + ] = CANONICAL_ORIGIN_BATTERY_METRIC_REGISTRY, + _canonical_joint_registry: Mapping[ + tuple[str, str, tuple[str, ...], int], str + ] = CANONICAL_ORIGIN_BATTERY_JOINT_METRIC_REGISTRY, +) -> None: + """Validate gate-specific receipts against the captured canonical doctrine.""" + + boundary = f"Gate {gate_name!r} manifest emission" + authority = details.get("authority") + if not isinstance(authority, Mapping): + raise ValueError( + f"{boundary}: no stacked authority receipt; production manifest " + "emission is forbidden." + ) + _validate_production_authority_receipt(authority, boundary=boundary) + authority_sha256 = authority["sha256"] + plan_sha256 = authority["components"]["gap_fill_plan"]["sha256"] + post_puf_surface_sha256 = authority["components"]["post_puf_transfer_surface"][ + "sha256" + ] + surface_sha256 = authority["components"]["declared_surface"]["sha256"] + expected_keys = _surface_target_keys(_canonical_surface) + + def reject(reason: str) -> None: + raise ValueError( + f"{boundary}: {reason}; production manifest emission is forbidden." + ) + + tail_support_receipt = details.get(_TAIL_SUPPORT_GATE_DETAIL_KEY) + if tail_support_receipt is not None: + if not isinstance(tail_support_receipt, Mapping): + reject("capital-gains-tail support receipt must be an object") + try: + validate_puf_capital_gains_tail_terminal_support_receipt( + tail_support_receipt + ) + except (TypeError, ValueError) as error: + reject(f"capital-gains-tail support receipt is invalid: {error}") + + def nonnegative_int( + receipt: Mapping[str, object], + field_name: str, + *, + label: str, + ) -> int: + value = receipt.get(field_name) + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + reject(f"{label} {field_name} must be a non-negative integer") + return value + + def validate_structural_absence_receipt( + raw_receipt: object, + *, + label: str, + battery: bool, + ) -> tuple[int, dict[str, int]]: + if not isinstance(raw_receipt, Mapping): + reject(f"{label} must carry canonical recipient-absence authority") + expected_fields = { + "rule_id", + "selection", + "reason", + "status", + "rows", + "by_origin_role", + "unexpected_null_rows", + "structural_rows_filled", + } + if battery: + expected_fields.update( + {"comparison_clone_index", "rows_excluded_from_scope"} + ) + if set(raw_receipt) != expected_fields: + reject(f"{label} structural-absence receipt schema mismatch") + if ( + raw_receipt.get("rule_id") != _ACS_GQ_RENT_ABSENCE_RULE_ID + or raw_receipt.get("selection") != _ACS_GQ_RENT_ABSENCE_SELECTION + or raw_receipt.get("reason") != _ACS_GQ_RENT_ABSENCE_REASON + or raw_receipt.get("status") != "exact_structural_absence" + ): + reject(f"{label} structural-absence doctrine mismatch") + rows = nonnegative_int(raw_receipt, "rows", label=label) + unexpected = nonnegative_int( + raw_receipt, + "unexpected_null_rows", + label=label, + ) + filled = nonnegative_int( + raw_receipt, + "structural_rows_filled", + label=label, + ) + raw_by_role = raw_receipt.get("by_origin_role") + if not isinstance(raw_by_role, Mapping): + reject(f"{label} structural by-origin-role counts are not a mapping") + by_role: dict[str, int] = {} + for cell, count in raw_by_role.items(): + cell_channel, separator, clone_role = str(cell).partition("/clone_") + if ( + not separator + or cell_channel != ACS_STACKED_SUPPORT_CHANNEL + or not clone_role.isdigit() + or isinstance(count, bool) + or not isinstance(count, int) + or count <= 0 + ): + reject(f"{label} structural by-origin-role receipt is malformed") + by_role[str(cell)] = count + if rows != sum(by_role.values()): + reject(f"{label} structural row count does not equal its role counts") + if passed and (unexpected != 0 or filled != 0): + reject(f"{label} passing structural-absence equation is not exact") + if battery: + if raw_receipt.get("comparison_clone_index") != 0: + reject(f"{label} structural battery clone scope must be clone 0") + excluded = nonnegative_int( + raw_receipt, + "rows_excluded_from_scope", + label=label, + ) + if excluded != by_role.get(f"{ACS_STACKED_SUPPORT_CHANNEL}/clone_0", 0): + reject( + f"{label} structural battery exclusion count differs from " + "its clone-0 authority" + ) + return rows, by_role + + if gate_name == _COMPLETENESS_GATE_NAME: + expected_labels = { + f"{entity}/{family}/{column}" + for entity, family, column, _clone_index in expected_keys + } + direction_by_label = { f"{entity}/{family}/{column}": direction for direction in _canonical_plan for entity, families in direction.target_families.items() @@ -3957,9 +7498,9 @@ def validate_structural_absence_receipt( or len(_canonical_early_transfer_keys) != 48 or len(_canonical_late_transfer_keys) != 70 or len(_canonical_late_puf_producer_keys) != 43 - or len(_canonical_late_source_producer_keys) != 30 + or len(_canonical_late_source_producer_keys) != 29 or len(_canonical_late_puf_producer_keys & _canonical_late_source_producer_keys) - != 3 + != 2 or _canonical_late_puf_producer_keys | _canonical_late_source_producer_keys != _canonical_late_transfer_keys or _canonical_early_transfer_keys & _canonical_late_transfer_keys @@ -3988,8 +7529,8 @@ def validate_structural_absence_receipt( "Canonical stacked authority must partition the exact 118-target " "transfer surface into 48 early gap-fill and 70 post-PUF targets " "inside an exact 131-target terminal surface and metric registry; " - "the late surface must be exactly covered by 43 PUF-clone and 30 " - "ASEC-source producer targets with their declared three-target overlap." + "the late surface must be exactly covered by 43 PUF-clone and 29 " + "ASEC-source producer targets with their declared two-target overlap." ) @@ -4665,6 +8206,87 @@ class StackedPostPufTransferResult: transfer_result: AcsTransferResult +@dataclass(frozen=True) +class StackedLateProducerResult: + """A fully executed late-producer DAG and its aggregate provenance.""" + + frame: Frame + receipt: Mapping[str, object] + primary_puf_result: StackedPufPassResult + source_completion_receipt: Mapping[str, object] + transfer_result: AcsTransferResult + transition_authority_sha256: str + + +def _producer_role_surface_for_group( + group_surface: TargetFamilies, + producer_surface: TargetFamilies, +) -> TargetFamilies: + """Project canonical producer roles onto one bounded transfer family.""" + + producer_targets = { + (entity, target) + for entity, families in producer_surface.items() + for targets in families.values() + for target in targets + } + return { + entity: { + family: tuple( + target for target in targets if (entity, target) in producer_targets + ) + } + for entity, families in group_surface.items() + for family, targets in families.items() + if any((entity, target) in producer_targets for target in targets) + } + + +def transfer_stacked_post_puf_group( + frame: Frame, + *, + group_name: str, + seed: int = 0, + n_estimators: int = 100, + max_targets_per_fit: int = DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, + target_bank: AcsTransferTargetBank | None = None, + execution_contract: Mapping[str, object] | None = None, +) -> StackedPostPufTransferResult: + """Execute one canonical bounded late-transfer producer.""" + + groups = {group.name: group for group in CANONICAL_US_LATE_TRANSFER_GROUPS} + if group_name not in groups: + raise ValueError( + f"Unknown canonical US late-transfer producer {group_name!r}; " + f"expected one of {sorted(groups)}." + ) + group = groups[group_name] + authority = _production_stacked_authority() + result = _transfer_stacked_post_puf_inputs_evaluate( + frame, + authority=authority, + production=True, + seed=seed, + n_estimators=n_estimators, + max_targets_per_fit=max_targets_per_fit, + target_bank=target_bank, + target_families=group.target_families, + derive_schedule_d=False, + execution_contract=execution_contract, + ) + return StackedPostPufTransferResult( + frame=result.frame, + receipt={ + **dict(result.receipt), + "producer": group.name, + "entity": group.entity, + "family": group.family, + "ordered_targets": list(group.targets), + }, + transfer_result=result.transfer_result, + ) + + def transfer_stacked_post_puf_inputs( frame: Frame, *, @@ -4683,6 +8305,9 @@ def transfer_stacked_post_puf_inputs( n_estimators=n_estimators, max_targets_per_fit=max_targets_per_fit, target_bank=target_bank, + target_families=None, + derive_schedule_d=True, + execution_contract=None, ) @@ -4706,6 +8331,9 @@ def _transfer_stacked_post_puf_inputs_with_test_authority( n_estimators=n_estimators, max_targets_per_fit=max_targets_per_fit, target_bank=target_bank, + target_families=None, + derive_schedule_d=True, + execution_contract=None, ) @@ -4718,6 +8346,9 @@ def _transfer_stacked_post_puf_inputs_evaluate( n_estimators: int, max_targets_per_fit: int, target_bank: AcsTransferTargetBank | None, + target_families: TargetFamilies | None, + derive_schedule_d: bool, + execution_contract: Mapping[str, object] | None, ) -> StackedPostPufTransferResult: """Run the late transfer from the one role carrying every declared target.""" @@ -4741,23 +8372,39 @@ def _transfer_stacked_post_puf_inputs_evaluate( frame, boundary="stacked post-PUF transfer attachment", ) - surface = authority.post_puf_transfer_surface + surface = ( + authority.post_puf_transfer_surface + if target_families is None + else target_families + ) if not _surface_target_keys(surface): raise ValueError("Stacked post-PUF transfer requires at least one target.") + puf_producer_surface = ( + authority.post_puf_puf_producer_surface + if target_families is None + else _producer_role_surface_for_group( + surface, + authority.post_puf_puf_producer_surface, + ) + ) + source_producer_surface = ( + authority.post_puf_source_producer_surface + if target_families is None + else _producer_role_surface_for_group( + surface, + authority.post_puf_source_producer_surface, + ) + ) pre_counts = _verify_post_puf_transfer_activation_authority( frame, target_families=surface, - puf_producer_families=authority.post_puf_puf_producer_surface, - source_producer_families=authority.post_puf_source_producer_surface, + puf_producer_families=puf_producer_surface, + source_producer_families=source_producer_surface, ) donor = _post_puf_donor_projection(frame) - puf_producer_keys = set( - _surface_target_keys(authority.post_puf_puf_producer_surface) - ) - source_producer_keys = set( - _surface_target_keys(authority.post_puf_source_producer_surface) - ) + puf_producer_keys = set(_surface_target_keys(puf_producer_surface)) + source_producer_keys = set(_surface_target_keys(source_producer_surface)) producer_snapshot = { (entity, target): _post_puf_producer_snapshot( frame, @@ -4774,17 +8421,20 @@ def _transfer_stacked_post_puf_inputs_evaluate( frame, donor, target_families=surface, + donor_spine=ASEC_PUF_DONOR_SPINE, donor_channel=None, seed=seed, n_estimators=n_estimators, max_targets_per_fit=max_targets_per_fit, target_bank=target_bank, + derive_schedule_d=derive_schedule_d, + execution_contract=execution_contract, ) target_receipts = _verify_post_puf_transfer_outcome( transfer.frame, target_families=surface, - puf_producer_families=authority.post_puf_puf_producer_surface, - source_producer_families=authority.post_puf_source_producer_surface, + puf_producer_families=puf_producer_surface, + source_producer_families=source_producer_surface, pre_counts=pre_counts, producer_snapshot=producer_snapshot, result=transfer, @@ -5063,20 +8713,859 @@ def _verify_post_puf_transfer_outcome( ("puf_clone", puf_produced), ("asec_source", source_produced), ) - if active - ], - "producer_rows": pre_counts[(entity, target)]["producer_rows"], - "authorized_null_rows": authorized, - "imputed_rows": imputed, - "unmodeled_rows": unmodeled, - "residual_null_rows": residual_nulls, + if active + ], + "producer_rows": pre_counts[(entity, target)]["producer_rows"], + "authorized_null_rows": authorized, + "imputed_rows": imputed, + "unmodeled_rows": unmodeled, + "residual_null_rows": residual_nulls, + } + if failures: + raise ValueError( + "Stacked post-PUF transfer outcome verification failed:\n " + + "\n ".join(failures) + ) + return target_receipts + + +def _late_required_scope_mask( + frame: Frame, + *, + entity: str, + required_scope: str, +) -> pd.Series: + """Resolve one declared late-stage row scope without inferring absence.""" + + table = frame.table(entity) + if required_scope == "whole_pool": + return pd.Series(True, index=table.index, dtype=bool) + if required_scope == "asec_source": + return ( + table[support_channel_column(entity)] + .astype(str) + .eq(BASE_ASEC_SUPPORT_CHANNEL) + ) + if required_scope == "acs_source": + return ( + table[support_channel_column(entity)] + .astype(str) + .eq(ACS_STACKED_SUPPORT_CHANNEL) + ) + if required_scope == "puf_clone": + return pd.to_numeric( + table[support_clone_index_column(entity)], + errors="raise", + ).gt(0) + raise ValueError(f"Unknown US late-producer scope {required_scope!r}.") + + +def _late_input_readiness_rows( + frame: Frame, + contract: ProducerContract, + *, + available_input_receipts: Mapping[str, Mapping[str, object]] | None = None, +) -> tuple[dict[ProducerInput, int], dict[ProducerInput, int]]: + """Count missing rows and invalid values as distinct readiness states.""" + + available = ( + {} if available_input_receipts is None else dict(available_input_receipts) + ) + unfilled: dict[ProducerInput, int] = {} + invalid: dict[ProducerInput, int] = {} + for requirement in contract.inputs: + column_states = { + input_column: _late_input_column_readiness_rows( + frame, + input_column=input_column, + required_scope=requirement.required_scope, + producer_name=contract.name, + available_input_receipts=available, + ) + for alternative in requirement.alternatives + for input_column in alternative + } + alternative_missing_counts = [ + sum(column_states[input_column][0] for input_column in alternative) + for alternative in requirement.alternatives + ] + # Invalid finite-numeric values never become absence merely because a + # different spelling is absent. Callbacks select alternatives by + # physical availability, so every present declared numeric column must + # be valid before the callback may inspect it. + unfilled[requirement] = min(alternative_missing_counts) + invalid[requirement] = sum( + column_states[input_column][1] for input_column in column_states + ) + return unfilled, invalid + + +def _late_unfilled_input_rows( + frame: Frame, + contract: ProducerContract, + *, + available_input_receipts: Mapping[str, Mapping[str, object]] | None = None, +) -> dict[ProducerInput, int]: + """Compatibility projection of the distinct late-input readiness state.""" + + unfilled, _invalid = _late_input_readiness_rows( + frame, + contract, + available_input_receipts=available_input_receipts, + ) + return unfilled + + +def _late_input_column_readiness_rows( + frame: Frame, + *, + input_column: ProducerInputColumn, + required_scope: str, + producer_name: str, + available_input_receipts: Mapping[str, Mapping[str, object]], +) -> tuple[int, int]: + """Return ``(missing_rows, invalid_values)`` for one declared column.""" + + if input_column.entity == "frame": + if not input_column.column.startswith("@"): + raise ValueError( + "Frame-level late inputs require an @ name; " + f"got {input_column.column!r}." + ) + metadata_key = input_column.column.removeprefix("@") + return ( + (0, 0) if isinstance(frame.metadata.get(metadata_key), Mapping) else (1, 0) + ) + table = frame.table(input_column.entity) + scope = _late_required_scope_mask( + frame, + entity=input_column.entity, + required_scope=required_scope, + ) + if input_column.column == "@resolved_weight": + weights = np.asarray( + frame.resolve_weights(input_column.entity).values, + dtype=np.float64, + ) + if weights.shape != (len(table),): + return 0, int(scope.sum()) + return 0, int((~np.isfinite(weights) & scope.to_numpy(dtype=bool)).sum()) + if input_column.column.startswith("@"): + receipt_key = f"{input_column.entity}.{input_column.column}" + receipt = available_input_receipts.get(receipt_key) + if _late_available_input_receipt_is_valid( + receipt, + producer=producer_name, + entity=input_column.entity, + column=input_column.column, + ): + return 0, 0 + return max(1, int(scope.sum())), 0 + if input_column.column not in table: + return int(scope.sum()), 0 + values = table[input_column.column] + if input_column.value_kind == "column_present": + return 0, 0 + missing = values.isna() + invalid = pd.Series(False, index=values.index, dtype=bool) + if input_column.value_kind == "finite_numeric": + numeric = pd.to_numeric(values, errors="coerce").to_numpy( + dtype=np.float64, + na_value=np.nan, + ) + invalid = (~missing) & ~np.isfinite(numeric) + return int((missing & scope).sum()), int((invalid & scope).sum()) + + +def _late_declared_absence_receipts( + contract: ProducerContract, + unfilled_rows: Mapping[ProducerInput, int], + *, + invalid_rows: Mapping[ProducerInput, int], +) -> dict[str, Mapping[str, object]]: + """Materialize only absences explicitly tolerated by the contract.""" + + receipts: dict[str, Mapping[str, object]] = {} + for requirement, rows in unfilled_rows.items(): + if ( + rows <= 0 + or invalid_rows.get(requirement, 0) > 0 + or not requirement.tolerated_absence_receipts + ): + continue + for receipt_id in requirement.tolerated_absence_receipts: + receipts[receipt_id] = { + "receipt_id": receipt_id, + "status": "declared_absence", + "entity": requirement.entity, + "column": requirement.column, + "required_scope": requirement.required_scope, + "rows": rows, + "producer": contract.name, + "reason": "optional availability-pattern input", + } + return receipts + + +def _assert_primary_puf_stage_complete(frame: Frame) -> None: + """Validate the already-executed root producer before DAG dispatch.""" + + contract = CANONICAL_US_LATE_PRODUCER_REGISTRY[US_LATE_PRIMARY_PUF_STAGE] + failures: list[str] = [] + for output in contract.outputs: + if output.entity == "frame": + metadata_key = output.column.removeprefix("@") + if not isinstance(frame.metadata.get(metadata_key), Mapping): + failures.append( + f"frame.{output.column}: metadata receipt absent on " + f"{output.coverage_scope}" + ) + continue + table = frame.table(output.entity) + scope = _late_required_scope_mask( + frame, + entity=output.entity, + required_scope=output.coverage_scope, + ) + if output.column == "@resolved_weight": + weights = np.asarray( + frame.resolve_weights(output.entity).values, + dtype=np.float64, + ) + if weights.shape != (len(table),): + failures.append( + f"{output.entity}.@resolved_weight: shape {weights.shape} " + f"does not match {(len(table),)}" + ) + else: + count = int( + ( + ~np.isfinite(weights) & scope.to_numpy(dtype=bool, copy=False) + ).sum() + ) + if count: + failures.append( + f"{output.entity}.@resolved_weight: {count} invalid " + f"row(s) on {output.coverage_scope}" + ) + continue + if output.column not in table: + failures.append( + f"{output.entity}.{output.column}: column absent on " + f"{output.coverage_scope}" + ) + continue + values = table[output.column] + missing = values.isna() + if pd.api.types.is_numeric_dtype(values.dtype): + missing |= ~np.isfinite( + pd.to_numeric(values, errors="coerce").to_numpy(dtype=np.float64) + ) + count = int((missing & scope).sum()) + if count: + failures.append( + f"{output.entity}.{output.column}: {count} unfilled row(s) on " + f"{output.coverage_scope}" + ) + if failures: + raise ValueError( + f"Late producer {US_LATE_PRIMARY_PUF_STAGE!r} is not complete:\n " + + "\n ".join(failures) + ) + + +def _materialize_stacked_acs_earnings_universe( + frame: Frame, + *, + execution_config: Mapping[str, object] | None = None, +) -> AcsPumsEarningsUniverseApplication: + """Run and bind the declared pre-primary ACS earnings-universe producer.""" + + resolved_config = ( + _late_acs_earnings_universe_execution_binding() + if execution_config is None + else _json_ready(execution_config) + ) + columns = resolved_config.get("ordered_mapped_columns") + if not isinstance(columns, list) or any( + not isinstance(column, str) or not column for column in columns + ): + raise ValueError("Late ACS earnings-universe execution config is malformed.") + application = apply_acs_pums_earnings_universe_zeros( + frame, + columns=tuple(columns), + boundary="late ACS PUMS earnings-universe producer", + ) + receipt = _json_ready(application.receipt) + contract = resolved_config.get("contract_identity") + if not isinstance(contract, Mapping) or any( + receipt.get(field) != contract.get(field) + for field in ( + "version", + "source_channel", + "minimum_age", + "aggregation", + "produced_frame_semantics", + ) + ): + raise ValueError( + "Late ACS earnings-universe application receipt differs from its " + "bound runtime contract." + ) + receipt_rules = [ + { + "rule_id": rule.get("rule_id"), + "source_column": rule.get("source_column"), + "mapped_column": rule.get("mapped_column"), + } + for rule in receipt.get("rules", {}).values() + ] + if receipt_rules != contract.get("rules"): + raise ValueError( + "Late ACS earnings-universe application rules differ from their " + "bound runtime contract." + ) + metadata_key = US_LATE_ACS_EARNINGS_UNIVERSE_RECEIPT_INPUT.removeprefix("@") + if metadata_key in application.frame.metadata: + raise ValueError( + "Late ACS PUMS earnings-universe producer found a pre-existing " + "application receipt." + ) + bound = Frame( + { + entity: application.frame.table(entity) + for entity in application.frame.entities + }, + application.frame.schema, + { + entity: application.frame.weights_for(entity) + for entity in application.frame.weighted_entities + }, + application.frame.strata, + mass_log=application.frame.mass_log, + metadata={**application.frame.metadata, metadata_key: receipt}, + ) + return AcsPumsEarningsUniverseApplication(bound, receipt) + + +def _aggregate_late_transfer_result( + frame: Frame, + *, + group_results: Mapping[ + str, + tuple[ + Mapping[str, object], + tuple[object, ...], + tuple[FitWeightRecord, ...], + tuple[str, ...], + str | None, + ], + ], + execution_order: Sequence[str], +) -> StackedPostPufTransferResult: + """Bind all bounded group outcomes into the canonical 70-target receipt.""" + + expected_groups = tuple(group.name for group in CANONICAL_US_LATE_TRANSFER_GROUPS) + if set(group_results) != set(expected_groups): + raise ValueError( + "US late-transfer finalization requires every canonical group once; " + f"missing={sorted(set(expected_groups) - set(group_results))}, " + f"extra={sorted(set(group_results) - set(expected_groups))}." + ) + authority = _production_stacked_authority() + canonical_family = { + (entity, target): family + for entity, families in authority.post_puf_transfer_surface.items() + for family, targets in families.items() + for target in targets + } + aggregate_targets: dict[str, object] = {} + imputed_inputs = [] + fit_records = [] + deferred_inputs: list[str] = [] + resolved_channels: set[str | None] = set() + group_receipts: dict[str, object] = {} + residual_null_rows = 0 + for group in CANONICAL_US_LATE_TRANSFER_GROUPS: + ( + group_receipt, + group_imputed_inputs, + group_fit_records, + group_deferred_inputs, + resolved_donor_channel, + ) = group_results[group.name] + if group_receipt.get("producer") != group.name: + raise ValueError( + f"US late-transfer group receipt for {group.name!r} is misbound." + ) + raw_targets = group_receipt.get("targets") + if not isinstance(raw_targets, Mapping): + raise ValueError( + f"US late-transfer group {group.name!r} has no target receipts." + ) + expected_target_labels = { + f"{group.entity}/{group.family}/{target}" for target in group.targets + } + if set(raw_targets) != expected_target_labels: + raise ValueError( + f"US late-transfer group {group.name!r} target receipt drift; " + f"expected={sorted(expected_target_labels)}, " + f"got={sorted(raw_targets)}." + ) + for target in group.targets: + bounded_label = f"{group.entity}/{group.family}/{target}" + family = canonical_family[(group.entity, target)] + aggregate_targets[f"{group.entity}/{family}/{target}"] = dict( + raw_targets[bounded_label] + ) + table = frame.table(group.entity) + if target not in table: + residual_null_rows += len(table) + else: + residual_null_rows += int(table[target].isna().sum()) + group_receipts[group.name] = dict(group_receipt) + imputed_inputs.extend(group_imputed_inputs) + fit_records.extend(group_fit_records) + deferred_inputs.extend(group_deferred_inputs) + resolved_channels.add(resolved_donor_channel) + if residual_null_rows: + raise ValueError( + "US late-transfer DAG finalization found " + f"{residual_null_rows} residual null target cell(s); zero are allowed." + ) + if len(resolved_channels) != 1: + raise ValueError( + "US late-transfer groups disagree on resolved donor channel: " + f"{sorted(map(str, resolved_channels))}." + ) + aggregate = AcsTransferResult( + frame=frame, + imputed_inputs=tuple(imputed_inputs), + fit_records=tuple(fit_records), + deferred_inputs=tuple(dict.fromkeys(deferred_inputs)), + resolved_donor_channel=next(iter(resolved_channels)), + ) + receipt = { + "authority": _authority_receipt(authority), + "producer_schedule": dict(us_late_producer_schedule_receipt()), + "producer_execution_order": list(execution_order), + "donor_selection": "owner_projection_of_asec_origin_clone_1", + "donor_channel": BASE_ASEC_SUPPORT_CHANNEL, + "donor_clone_index": PUF_TAX_DETAIL_CLONE_INDEX, + "recipient_selection": ("target_specific_complement_of_declared_producer_rows"), + "resolved_donor_channel": aggregate.resolved_donor_channel, + "groups": group_receipts, + "targets": aggregate_targets, + "fit_records": [ + {"fit_name": record.fit_name, "weight_kind": record.weight_kind} + for record in aggregate.fit_records + ], + "completion": { + "status": "complete", + "group_count": len(expected_groups), + "target_count": len(aggregate_targets), + "residual_null_rows": 0, + }, + } + validate_stacked_post_puf_transfer_receipt( + receipt, + boundary="US late-transfer DAG finalization", + ) + return StackedPostPufTransferResult(frame, receipt, aggregate) + + +def run_stacked_late_producer_dag( + frame: Frame, + *, + primary_puf_producer: Callable[[Frame], StackedPufPassResult], + primary_resource_receipts: Mapping[str, Mapping[str, object]], + seed: int = 0, + n_estimators: int = 100, + max_targets_per_fit: int = DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, + target_banks: Mapping[str, AcsTransferTargetBank | None] | None = None, + absence_receipts: Mapping[str, Mapping[str, object]] | None = None, +) -> StackedLateProducerResult: + """Derive and execute the complete late stage from producer contracts.""" + + from microcosm.build.us_runtime.multispine_pool import ( + POOL_POST_CLONE_SOURCE_OPERATOR_ORDER, + finalize_multispine_source_inputs, + run_multispine_post_clone_source_operator, + ) + + if max_targets_per_fit != DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT: + raise ValueError( + "Canonical US late-producer groups require " + f"max_targets_per_fit={DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT}; " + f"got {max_targets_per_fit}." + ) + if not callable(primary_puf_producer): + raise TypeError("US late-producer DAG requires a primary-PUF callback.") + if not isinstance(primary_resource_receipts, Mapping): + raise TypeError( + "US late-producer DAG primary resource receipts must be a mapping." + ) + expected_primary_resources = _late_contract_available_input_keys( + CANONICAL_US_LATE_PRODUCER_REGISTRY[US_LATE_PRIMARY_PUF_STAGE] + ) + if set(primary_resource_receipts) != expected_primary_resources: + raise ValueError( + "US late-producer DAG primary resource receipts must exactly cover " + "the declared virtual inputs; " + f"missing={sorted(expected_primary_resources - set(primary_resource_receipts))}, " + f"extra={sorted(set(primary_resource_receipts) - expected_primary_resources)}." + ) + if US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY in frame.metadata: + raise ValueError( + "US late-producer DAG entry already carries a transition authority; " + "refusing to execute the transition twice." + ) + expected_groups = {group.name for group in CANONICAL_US_LATE_TRANSFER_GROUPS} + banks = {} if target_banks is None else dict(target_banks) + if target_banks is not None and set(banks) != expected_groups: + raise ValueError( + "US late-producer target-bank mapping must exactly cover the " + f"canonical groups; missing={sorted(expected_groups - set(banks))}, " + f"extra={sorted(set(banks) - expected_groups)}." + ) + declared_absence = {} if absence_receipts is None else dict(absence_receipts) + current = frame + input_frame_sha256 = _late_frame_content_sha256(frame) + schedule_receipt = _json_ready(us_late_producer_schedule_receipt()) + previous_execution_sha256 = _late_execution_genesis_sha256( + producer_schedule_sha256=schedule_receipt["payload_sha256"], + input_frame_sha256=input_frame_sha256, + ) + execution_order: list[str] = [] + execution_receipts: list[dict[str, object]] = [] + primary_puf_result: StackedPufPassResult | None = None + source_receipts: dict[str, Mapping[str, object]] = {} + source_completion_receipt: Mapping[str, object] | None = None + group_results: dict[ + str, + tuple[ + Mapping[str, object], + tuple[object, ...], + tuple[FitWeightRecord, ...], + tuple[str, ...], + str | None, + ], + ] = {} + group_by_name = {group.name: group for group in CANONICAL_US_LATE_TRANSFER_GROUPS} + for schedule_index, producer_name in enumerate( + CANONICAL_US_LATE_PRODUCER_SCHEDULE.order + ): + contract = CANONICAL_US_LATE_PRODUCER_REGISTRY[producer_name] + if producer_name == US_LATE_ACS_EARNINGS_UNIVERSE_STAGE: + node_available_inputs = _late_acs_earnings_universe_resource_receipts() + elif producer_name == US_LATE_PRIMARY_PUF_STAGE: + node_available_inputs = dict(primary_resource_receipts) + elif contract.kind == "post_clone_source": + node_available_inputs = _late_source_resource_receipts( + producer_name=producer_name, + ) + elif producer_name == US_LATE_SOURCE_FINALIZER_STAGE: + node_available_inputs = _late_source_finalizer_resource_receipts() + node_available_inputs.update( + { + f"person.@source_receipt:{operator}": ( + _late_available_input_receipt( + producer=US_LATE_SOURCE_FINALIZER_STAGE, + entity="person", + column=f"@source_receipt:{operator}", + rows=len(current.table("person")), + binding={ + "resource_kind": "source_operator_receipt", + "schema_version": 1, + "source_operator": operator, + "source_receipt_sha256": _canonical_sha256( + _json_ready(source_receipts[operator]) + ), + }, + ) + ) + for operator in POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + if operator in source_receipts } - if failures: - raise ValueError( - "Stacked post-PUF transfer outcome verification failed:\n " - + "\n ".join(failures) + ) + elif contract.kind == "late_transfer": + group = group_by_name[producer_name] + node_available_inputs = _late_transfer_resource_receipts( + group_name=group.name, + entity=group.entity, + family=group.family, + targets=group.targets, + seed=seed, + n_estimators=n_estimators, + max_targets_per_fit=max_targets_per_fit, + target_bank=banks.get(producer_name), + ) + else: + node_available_inputs = {} + unfilled_rows, invalid_rows = _late_input_readiness_rows( + current, + contract, + available_input_receipts=node_available_inputs, ) - return target_receipts + node_absence_receipts = _late_declared_absence_receipts( + contract, + unfilled_rows, + invalid_rows=invalid_rows, + ) + for receipt_id, receipt in node_absence_receipts.items(): + previous = declared_absence.setdefault(receipt_id, receipt) + if dict(previous) != dict(receipt): + raise ValueError( + f"Late producer {producer_name!r} absence receipt " + f"{receipt_id!r} conflicts with supplied evidence." + ) + declared_input_evidence = _late_declared_input_evidence( + current, + contract, + available_input_receipts=node_available_inputs, + unfilled_rows=unfilled_rows, + invalid_rows=invalid_rows, + ) + outcome: dict[str, object] = {} + + def execute( + *, + bound_contract: ProducerContract = contract, + bound_producer_name: str = producer_name, + bound_frame: Frame = current, + bound_outcome: dict[str, object] = outcome, + bound_available_inputs: Mapping[ + str, Mapping[str, object] + ] = node_available_inputs, + ) -> None: + if bound_contract.kind == "acs_earnings_universe": + config_receipt = bound_available_inputs[ + f"person.{US_LATE_ACS_EARNINGS_UNIVERSE_CONFIG_INPUT}" + ] + result = _materialize_stacked_acs_earnings_universe( + bound_frame, + execution_config=config_receipt["binding"], + ) + elif bound_contract.kind == "primary_puf": + result = primary_puf_producer(bound_frame) + elif bound_contract.kind == "post_clone_source": + operator = bound_producer_name.removeprefix("source:") + result = run_multispine_post_clone_source_operator( + bound_frame, + operator, + ) + elif bound_contract.kind == "source_finalizer": + result = finalize_multispine_source_inputs( + bound_frame, + operator_receipts=source_receipts, + ) + elif bound_contract.kind == "late_transfer": + group = group_by_name[bound_producer_name] + config_receipt = bound_available_inputs[ + f"{group.entity}.{US_LATE_TRANSFER_MODEL_CONFIG_INPUT}" + ] + config_binding = config_receipt["binding"] + assert isinstance(config_binding, Mapping) + execution_contract = config_binding["transfer_execution_contract"] + assert isinstance(execution_contract, Mapping) + result = transfer_stacked_post_puf_group( + bound_frame, + group_name=bound_producer_name, + seed=seed, + n_estimators=n_estimators, + max_targets_per_fit=max_targets_per_fit, + target_bank=banks.get(bound_producer_name), + execution_contract=execution_contract, + ) + else: + raise AssertionError( + f"Unhandled US late-producer kind {bound_contract.kind!r}." + ) + bound_outcome["result"] = result + + run_producer_when_ready( + contract, + execute, + unfilled_rows=unfilled_rows, + invalid_rows=invalid_rows, + absence_receipts=declared_absence, + ) + result = outcome["result"] + if contract.kind == "acs_earnings_universe": + _assert_late_callback_consumed_bound_config( + producer=producer_name, + entity="person", + column=US_LATE_ACS_EARNINGS_UNIVERSE_CONFIG_INPUT, + available_input_receipts=node_available_inputs, + actual_binding=_late_acs_earnings_universe_execution_binding(), + ) + elif contract.kind == "post_clone_source": + _assert_late_callback_consumed_bound_config( + producer=producer_name, + entity="person", + column=US_LATE_SOURCE_EXECUTION_CONFIG_INPUT, + available_input_receipts=node_available_inputs, + actual_binding=_late_source_execution_config_binding(producer_name), + ) + elif contract.kind == "source_finalizer": + _assert_late_callback_consumed_bound_config( + producer=producer_name, + entity="person", + column=US_LATE_SOURCE_FINALIZER_EXECUTION_CONFIG_INPUT, + available_input_receipts=node_available_inputs, + actual_binding=_late_source_finalizer_execution_binding(), + ) + elif contract.kind == "late_transfer": + group = group_by_name[producer_name] + _assert_late_callback_consumed_bound_config( + producer=producer_name, + entity=group.entity, + column=US_LATE_TRANSFER_MODEL_CONFIG_INPUT, + available_input_receipts=node_available_inputs, + actual_binding=_late_transfer_model_config_binding( + group_name=group.name, + entity=group.entity, + family=group.family, + targets=group.targets, + seed=seed, + n_estimators=n_estimators, + max_targets_per_fit=max_targets_per_fit, + ), + ) + _assert_late_callback_consumed_bound_config( + producer=producer_name, + entity=group.entity, + column=US_LATE_TRANSFER_TARGET_BANK_INPUT, + available_input_receipts=node_available_inputs, + actual_binding=_late_transfer_target_bank_binding( + group_name=group.name, + target_bank=banks.get(producer_name), + ), + ) + current = result.frame + _validate_late_callback_output_metric_families(current, contract) + producer_receipt = _json_ready(result.receipt) + if contract.kind == "primary_puf": + _validate_primary_callback_resource_binding( + producer_receipt, + available_input_receipts=node_available_inputs, + boundary=f"late producer {producer_name!r}", + ) + output_surface = [ + _late_output_column_evidence( + current, + output=output, + producer_receipt=producer_receipt, + ) + for output in contract.outputs + ] + absent_outputs = [ + f"{item['entity']}.{item['column']}" + for item in output_surface + if item.get("status") != "present" + ] + if absent_outputs: + raise ValueError( + f"Late producer {producer_name!r} completed without declared " + f"output(s) {absent_outputs}." + ) + execution_row: dict[str, object] = { + "execution_index": schedule_index, + "producer": producer_name, + "kind": contract.kind, + "declared_inputs": declared_input_evidence, + "declared_absence_receipts": { + receipt_id: dict(receipt) + for receipt_id, receipt in node_absence_receipts.items() + }, + "available_input_receipts": { + receipt_id: dict(receipt) + for receipt_id, receipt in sorted(node_available_inputs.items()) + }, + "input_surface_sha256": _canonical_sha256(declared_input_evidence), + "output_surface": output_surface, + "output_surface_sha256": _canonical_sha256(output_surface), + "producer_receipt": producer_receipt, + "producer_receipt_sha256": _canonical_sha256(producer_receipt), + "previous_execution_sha256": previous_execution_sha256, + "status": "complete", + } + execution_row["sha256"] = _canonical_sha256(execution_row) + previous_execution_sha256 = str(execution_row["sha256"]) + execution_receipts.append(execution_row) + if contract.kind == "acs_earnings_universe": + execution_order.append(producer_name) + continue + if contract.kind == "primary_puf": + if not isinstance(result, StackedPufPassResult): + raise TypeError( + "US primary-PUF producer callback must return StackedPufPassResult." + ) + _assert_primary_puf_stage_complete(current) + primary_puf_result = result + continue + + execution_order.append(producer_name) + if contract.kind == "post_clone_source": + operator = producer_name.removeprefix("source:") + source_receipts[operator] = result.receipt + elif contract.kind == "source_finalizer": + source_completion_receipt = result.receipt + else: + group = group_by_name[producer_name] + if tuple(result.receipt.get("ordered_targets", ())) != group.targets: + raise ValueError( + f"Late-transfer producer {producer_name!r} changed its " + "declared target order." + ) + group_results[producer_name] = ( + result.receipt, + result.transfer_result.imputed_inputs, + result.transfer_result.fit_records, + result.transfer_result.deferred_inputs, + result.transfer_result.resolved_donor_channel, + ) + if source_completion_receipt is None: + raise AssertionError("US late-producer DAG did not finalize source inputs.") + if primary_puf_result is None: + raise AssertionError("US late-producer DAG did not execute primary PUF.") + aggregate = _aggregate_late_transfer_result( + current, + group_results=group_results, + execution_order=execution_order, + ) + late_receipt: dict[str, object] = { + "version": US_LATE_PRODUCER_RECEIPT_SCHEMA_VERSION, + "producer_schedule": schedule_receipt, + "input_frame_sha256": input_frame_sha256, + "output_frame_sha256": _late_frame_content_sha256(aggregate.frame), + "execution_chain_sha256": previous_execution_sha256, + "execution": execution_receipts, + "source_completion": _json_ready(source_completion_receipt), + "post_puf_transfer": _json_ready(aggregate.receipt), + } + late_receipt["sha256"] = _canonical_sha256(late_receipt) + validate_stacked_late_producer_receipt( + late_receipt, + boundary="US late-producer DAG finalization", + ) + authorized_frame, transition_authority_sha256 = ( + _bind_late_producer_transition_authority(aggregate.frame, late_receipt) + ) + validate_stacked_late_producer_receipt( + late_receipt, + boundary="US late-producer DAG live-output finalization", + frame=authorized_frame, + expected_transition_authority_sha256=transition_authority_sha256, + ) + return StackedLateProducerResult( + frame=authorized_frame, + receipt=late_receipt, + primary_puf_result=primary_puf_result, + source_completion_receipt=source_completion_receipt, + transfer_result=aggregate.transfer_result, + transition_authority_sha256=transition_authority_sha256, + ) # --------------------------------------------------------------------------- @@ -5084,6 +9573,156 @@ def _verify_post_puf_transfer_outcome( # --------------------------------------------------------------------------- +def us_puf_s_corp_universe_zero_rule_identity() -> dict[str, object]: + """Return the declared whole-pool meaning of the PUF S-corp leaf.""" + + return { + "rule_id": US_PUF_S_CORP_UNIVERSE_ZERO_RULE_ID, + "schema_version": 1, + "entity": "person", + "column": "s_corp_income", + "coverage_scope": "whole_pool", + "materialized_value": 0.0, + "source_semantics": ( + "puf_combined_partnership_s_corp_carried_by_partnership_income" + ), + "donor_precondition": "finite_exact_zero", + "puf_clone_precondition": "finite_exact_zero", + "native_precondition": "all_null", + "assignment": "explicit_array_assignment", + } + + +def _materialize_us_puf_s_corp_universe_zero( + frame: Frame, + donor_tax_units: pd.DataFrame, +) -> tuple[Frame, dict[str, object]]: + """Materialize the certified all-zero S-corp leaf over the whole pool. + + The PUF source stores combined partnership/S-corporation income in + ``partnership_income`` and exposes ``s_corp_income`` as an exact-zero + schema leaf. The QRF therefore proves the clone values, while this owner + extends that declared universe meaning to native stacked rows. Every + precondition is checked before one explicit array assignment; this is not + a missing-value fallback. + """ + + column = "s_corp_income" + if column not in donor_tax_units: + raise ValueError( + "US PUF s_corp_income universe-zero materialization requires the " + "declared donor column." + ) + donor_values = pd.to_numeric(donor_tax_units[column], errors="coerce").to_numpy( + dtype=np.float64, + na_value=np.nan, + ) + donor_nonfinite = int((~np.isfinite(donor_values)).sum()) + if donor_nonfinite: + raise ValueError( + "US PUF s_corp_income universe-zero donor precondition failed: " + f"{donor_nonfinite} nonfinite value(s)." + ) + donor_nonzero = int((donor_values != 0.0).sum()) + if donor_nonzero: + raise ValueError( + "US PUF s_corp_income universe-zero donor precondition failed: " + f"{donor_nonzero} nonzero value(s)." + ) + + person_entity = frame.schema.person_entity + person = frame.table(person_entity).copy(deep=True) + if column not in person: + raise ValueError( + "US PUF s_corp_income universe-zero materialization requires the " + "primary-QRF output column." + ) + clone_column = support_clone_index_column(person_entity) + clone_values = pd.to_numeric(person[clone_column], errors="coerce").to_numpy( + dtype=np.float64, + na_value=np.nan, + ) + if not np.isfinite(clone_values).all(): + raise ValueError( + "US PUF s_corp_income universe-zero materialization found a " + "nonfinite person clone role." + ) + native = clone_values == 0.0 + produced = clone_values > 0.0 + preexisting_native = int(person.loc[native, column].notna().sum()) + if preexisting_native: + raise ValueError( + "US PUF s_corp_income universe-zero native precondition failed: " + f"{preexisting_native} native cell(s) were already materialized." + ) + + produced_values = pd.to_numeric( + person.loc[produced, column], + errors="coerce", + ).to_numpy(dtype=np.float64, na_value=np.nan) + produced_nonfinite = int((~np.isfinite(produced_values)).sum()) + if produced_nonfinite: + raise ValueError( + "US PUF s_corp_income universe-zero clone precondition failed: " + f"{produced_nonfinite} nonfinite clone/tail value(s)." + ) + produced_nonzero = int((produced_values != 0.0).sum()) + if produced_nonzero: + raise ValueError( + "US PUF s_corp_income universe-zero clone precondition failed: " + f"{produced_nonzero} nonzero clone/tail value(s)." + ) + + output_values = pd.to_numeric(person[column], errors="coerce").to_numpy( + dtype=np.float64, + na_value=np.nan, + copy=True, + ) + output_values[native] = np.zeros(int(native.sum()), dtype=np.float64) + person[column] = output_values + output_nonfinite = int((~np.isfinite(output_values)).sum()) + output_nonzero = int((output_values != 0.0).sum()) + if output_nonfinite or output_nonzero: + raise AssertionError( + "US PUF s_corp_income universe-zero explicit assignment did not " + "produce an exact finite-zero whole-pool column." + ) + + tables = {entity: frame.table(entity) for entity in frame.entities} + tables[person_entity] = person + materialized = Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + role_counts = { + str(int(role)): int((clone_values == role).sum()) + for role in sorted(set(clone_values)) + } + receipt: dict[str, object] = { + "rule": us_puf_s_corp_universe_zero_rule_identity(), + "status": "materialized", + "donor_rows_verified": len(donor_values), + "native_rows_materialized": int(native.sum()), + "produced_rows_verified": int(produced.sum()), + "person_rows": len(person), + "person_rows_by_clone_role": role_counts, + "post_materialization_nonfinite_rows": output_nonfinite, + "post_materialization_nonzero_rows": output_nonzero, + "donor_values_sha256": _late_table_values_sha256( + donor_tax_units.loc[:, [column]] + ), + "person_values_sha256": _late_table_values_sha256( + person.loc[:, [clone_column, column]] + ), + } + receipt["sha256"] = _canonical_sha256(receipt) + return materialized, receipt + + @dataclass(frozen=True) class StackedPufPassResult: """The post-PUF stacked frame plus attachment and fit receipts.""" @@ -5106,6 +9745,7 @@ def run_stacked_puf_pass( fit_records: list[FitWeightRecord] | None = None, tail_bound_diagnostics: list[dict[str, object]] | None = None, primary_qrf_checkpoint_dir: str | Path | None = None, + primary_qrf_input_binding: Mapping[str, object] | None = None, ) -> StackedPufPassResult: """Run the resumable primary QRF and clone-2 tail over the stacked spine. @@ -5131,6 +9771,7 @@ def run_stacked_puf_pass( fit_records=fit_records, tail_bound_diagnostics=tail_bound_diagnostics, primary_qrf_checkpoint_dir=primary_qrf_checkpoint_dir, + primary_qrf_input_binding=primary_qrf_input_binding, apply_capital_gains_tail=True, ) @@ -5164,10 +9805,44 @@ def _run_stacked_puf_pass_evaluate( fit_records: list[FitWeightRecord] | None = None, tail_bound_diagnostics: list[dict[str, object]] | None = None, primary_qrf_checkpoint_dir: str | Path | None = None, + primary_qrf_input_binding: Mapping[str, object] | None = None, apply_capital_gains_tail: bool, ) -> StackedPufPassResult: """Internal evaluator with one explicit fixture-only tail seam.""" + if primary_qrf_input_binding is not None: + noncanonical = { + "predictors": predictors, + "person_outputs": person_outputs, + "tax_unit_outputs": tax_unit_outputs, + } + explicit = sorted( + name for name, value in noncanonical.items() if value is not None + ) + if explicit: + raise ValueError( + "Stacked production primary QRF requires the import-declared " + f"canonical predictor/output surface; explicit={explicit}." + ) + missing_sinks = [ + name + for name, value in { + "fit_records": fit_records, + "tail_bound_diagnostics": tail_bound_diagnostics, + }.items() + if value is None + ] + if missing_sinks: + raise ValueError( + "Stacked production primary QRF requires its declared audit " + f"sink(s): {missing_sinks}." + ) + canonical_tail_bounds = ( + puf_tax_detail_tail_bound_quantiles_identity() + if person_outputs is None and tax_unit_outputs is None + else None + ) + tail_spec, tail_agi_bands = resolve_puf_capital_gains_tail_execution_inputs() validate_stacked_spine_frame(frame, boundary="stacked PUF pass entry") person_clone = frame.table("person")[support_clone_index_column("person")] if not person_clone.eq(0).all(): @@ -5175,12 +9850,31 @@ def _run_stacked_puf_pass_evaluate( "The stacked PUF pass owns clone attachment; found nonzero person " "support clone indices on its input." ) - universe_application = apply_acs_pums_earnings_universe_zeros( + universe_metadata_key = US_LATE_ACS_EARNINGS_UNIVERSE_RECEIPT_INPUT.removeprefix( + "@" + ) + universe_application_receipt = frame.metadata.get(universe_metadata_key) + if not isinstance(universe_application_receipt, Mapping): + raise ValueError( + "Stacked PUF pass requires the declared " + f"{US_LATE_ACS_EARNINGS_UNIVERSE_STAGE!r} producer receipt before " + "the primary callback may run." + ) + resolved_universe = resolve_acs_pums_earnings_universe( frame, - boundary="stacked PUF pass ACS earnings universe", + columns=tuple(ACS_PUMS_EARNINGS_SOURCE_COLUMNS), + boundary="stacked PUF pass ACS earnings-universe receipt", ) + if _json_ready(universe_application_receipt) != _json_ready( + resolved_universe.receipt + ): + raise ValueError( + "Stacked PUF pass ACS earnings-universe receipt does not match " + "the live produced frame." + ) cloned = clone_us_frame_for_puf_support( - universe_application.frame, + frame, + channels=(BASE_ASEC_SUPPORT_CHANNEL, PUF_TAX_DETAIL_SUPPORT_CHANNEL), clone_attachment_fraction=clone_attachment_fraction, clone_attachment_seed=clone_attachment_seed, ) @@ -5198,7 +9892,13 @@ def _run_stacked_puf_pass_evaluate( kwargs["person_outputs"] = tuple(person_outputs) if tax_unit_outputs is not None: kwargs["tax_unit_outputs"] = tuple(tax_unit_outputs) + primary_resource_receipts_sha256: str | None = None if primary_qrf_checkpoint_dir is None: + if primary_qrf_input_binding is not None: + raise ValueError( + "Stacked monolithic primary QRF cannot carry a checkpoint-input " + "binding without a checkpoint directory." + ) predictor_universe_receipts: list[dict[str, object]] = [] imputed = impute_us_puf_tax_detail_support( cloned, @@ -5207,6 +9907,7 @@ def _run_stacked_puf_pass_evaluate( n_estimators=n_estimators, fit_records=fit_records, tail_bound_diagnostics=tail_bound_diagnostics, + tail_bound_quantiles=canonical_tail_bounds, predictor_universe_receipts=predictor_universe_receipts, require_complete_recipient_predictors=True, absent_cells=PUF_ABSENT_CELLS_PRESERVE_NULLS, @@ -5223,9 +9924,78 @@ def _run_stacked_puf_pass_evaluate( "recipient_predictor_universe": predictor_universe_receipts[0], } else: + _validate_stacked_late_primary_checkpoint_input_binding( + primary_qrf_input_binding, + boundary="stacked primary-QRF checkpoint entry", + ) + assert isinstance(primary_qrf_input_binding, Mapping) + normalized_input_binding = _json_ready(primary_qrf_input_binding) + bound_resources = normalized_input_binding["primary_resource_receipts"] + assert isinstance(bound_resources, Mapping) + checkpoint_resource = bound_resources["tax_unit.@primary_qrf_checkpoint"] + assert isinstance(checkpoint_resource, Mapping) + checkpoint_binding = checkpoint_resource["binding"] + assert isinstance(checkpoint_binding, Mapping) checkpoint_dir = Path(primary_qrf_checkpoint_dir) + bound_checkpoint_identity = str( + checkpoint_binding["checkpoint_identity_sha256"] + ) + observed_checkpoint_identity = checkpoint_dir.resolve().name + if observed_checkpoint_identity != bound_checkpoint_identity: + raise ValueError( + "Stacked primary-QRF checkpoint directory identity differs " + "from its bound late-producer resource: " + f"directory={observed_checkpoint_identity!r}, " + f"bound={bound_checkpoint_identity!r}." + ) + actual_resources = stacked_late_primary_resource_receipts( + donor_tax_units, + primary_qrf_checkpoint_identity_sha256=observed_checkpoint_identity, + clone_attachment_fraction=clone_attachment_fraction, + clone_attachment_seed=clone_attachment_seed, + seed=seed, + n_estimators=n_estimators, + fit_records_enabled=fit_records is not None, + tail_bound_diagnostics_enabled=(tail_bound_diagnostics is not None), + predictors=predictors, + person_outputs=person_outputs, + tax_unit_outputs=tax_unit_outputs, + capital_gains_tail_spec=tail_spec, + capital_gains_tail_agi_bands=tail_agi_bands, + ) + if _json_ready(actual_resources) != _json_ready(bound_resources): + raise ValueError( + "Stacked primary-QRF callback invocation disagrees with its " + "declared late-producer donor/config resources." + ) + primary_resource_receipts_sha256 = _canonical_sha256(bound_resources) manifest_path = checkpoint_dir / PRIMARY_QRF_MANIFEST_FILENAME + input_binding_path = checkpoint_dir / _LATE_PRIMARY_QRF_INPUT_BINDING_FILENAME if manifest_path.exists(): + if not input_binding_path.is_file(): + raise ValueError( + "Stacked primary QRF checkpoint has no late-producer input " + f"binding: {input_binding_path}." + ) + try: + observed_input_binding = json.loads( + input_binding_path.read_text(encoding="utf-8") + ) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise ValueError( + "Stacked primary QRF checkpoint input binding is unreadable: " + f"{input_binding_path}." + ) from exc + _validate_stacked_late_primary_checkpoint_input_binding( + observed_input_binding, + boundary="stacked primary-QRF checkpoint resume", + ) + if observed_input_binding != normalized_input_binding: + raise ValueError( + "Stacked primary QRF checkpoint input binding differs from " + "the live late-producer donor/config resources; refusing " + "stale predictions." + ) resume_status = "resumed" else: if checkpoint_dir.exists() and any(checkpoint_dir.iterdir()): @@ -5243,15 +10013,27 @@ def _run_stacked_puf_pass_evaluate( absent_cells=PUF_ABSENT_CELLS_PRESERVE_NULLS, **kwargs, ) + input_binding_bytes = json.dumps( + normalized_input_binding, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + temporary_binding_path = input_binding_path.with_name( + f".{input_binding_path.name}.tmp" + ) + temporary_binding_path.write_bytes(input_binding_bytes) + temporary_binding_path.replace(input_binding_path) resume_status = "initialized" predictor_universe_receipt = ( primary_puf_qrf_recipient_predictor_universe_receipt(checkpoint_dir) ) - run_primary_puf_qrf_chain(checkpoint_dir) + run_primary_puf_qrf_chain(checkpoint_dir, environment={}) imputed, weight_kind = finalize_primary_puf_qrf_chain( cloned, checkpoint_dir, tail_bound_diagnostics=tail_bound_diagnostics, + tail_bound_quantiles=canonical_tail_bounds, ) if fit_records is not None: fit_records.append(FitWeightRecord(US_PUF_SUPPORT_FIT_NAME, weight_kind)) @@ -5259,6 +10041,7 @@ def _run_stacked_puf_pass_evaluate( "mode": "checkpoint_chain", "resume_status": resume_status, "checkpoint_manifest": str(manifest_path.resolve()), + "input_binding_sha256": normalized_input_binding["sha256"], "recipient_predictor_universe": predictor_universe_receipt, } @@ -5275,6 +10058,8 @@ def _run_stacked_puf_pass_evaluate( imputed, donor_tax_units, seed=seed, + spec=tail_spec, + agi_bands=tail_agi_bands, ) validate_puf_capital_gains_tail_manifest(tail_receipt) # The tail producer creates clone role 2 before its final origin @@ -5313,6 +10098,29 @@ def _run_stacked_puf_pass_evaluate( output = imputed tail_receipt = None tail_status = "fixture_only_skipped" + + declared_person_outputs = ( + tuple(PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS) + if person_outputs is None + else tuple(person_outputs) + ) + s_corp_rule_requested = "s_corp_income" in declared_person_outputs + if s_corp_rule_requested and ( + apply_capital_gains_tail or person_outputs is not None + ): + output, s_corp_universe_receipt = _materialize_us_puf_s_corp_universe_zero( + output, donor_tax_units + ) + else: + s_corp_universe_receipt = { + "rule": us_puf_s_corp_universe_zero_rule_identity(), + "status": ( + "fixture_only_skipped" + if s_corp_rule_requested + else "output_not_requested" + ), + } + s_corp_universe_receipt["sha256"] = _canonical_sha256(s_corp_universe_receipt) validate_stacked_spine_frame(output, boundary="stacked PUF pass output") person = output.table("person") @@ -5322,23 +10130,25 @@ def _run_stacked_puf_pass_evaluate( origin: int((channel.eq(origin) & clone_index.eq(1)).sum()) for origin in sorted(channel.unique()) } - return StackedPufPassResult( - frame=output, - receipt={ - "acs_earnings_universe_application": _json_ready( - universe_application.receipt - ), - "clone_attachment": _json_ready(attachment), - "doctrines": { - "require_complete_recipient_predictors": True, - "absent_cells": PUF_ABSENT_CELLS_PRESERVE_NULLS, + receipt: dict[str, object] = { + "acs_earnings_universe_application": _json_ready(universe_application_receipt), + "clone_attachment": _json_ready(attachment), + "doctrines": { + "require_complete_recipient_predictors": True, + "absent_cells": PUF_ABSENT_CELLS_PRESERVE_NULLS, + "whole_pool_output_universes": { + "person.s_corp_income": (us_puf_s_corp_universe_zero_rule_identity()), }, - "primary_puf_qrf": primary_qrf_receipt, - "puf_capital_gains_tail_transfer": tail_receipt, - "tail_status": tail_status, - "recipient_person_rows_by_origin": recipients_by_origin, }, - ) + "primary_puf_qrf": primary_qrf_receipt, + "puf_capital_gains_tail_transfer": tail_receipt, + "s_corp_income_universe_zero": s_corp_universe_receipt, + "tail_status": tail_status, + "recipient_person_rows_by_origin": recipients_by_origin, + } + if primary_resource_receipts_sha256 is not None: + receipt["primary_resource_receipts_sha256"] = primary_resource_receipts_sha256 + return StackedPufPassResult(frame=output, receipt=receipt) def _bind_stacked_tail_origin_receipt( @@ -5370,6 +10180,16 @@ def _bind_stacked_tail_origin_receipt( bound = _json_ready(tail_manifest) bound.pop("manifest_sha256", None) + overlap_ownership = _json_ready(us_late_overlap_ownership_receipt()) + existing_overlap_ownership = bound.get("late_overlap_ownership") + if ( + existing_overlap_ownership is not None + and existing_overlap_ownership != overlap_ownership + ): + raise ValueError( + "Stacked tail overlap ownership conflicts with the canonical owner matrix." + ) + bound["late_overlap_ownership"] = overlap_ownership clone_receipt = bound.get("clone") if not isinstance(clone_receipt, dict): raise ValueError("Stacked tail clone provenance receipt is malformed.") @@ -5435,6 +10255,19 @@ def assert_stacked_tail_cells_preserved( validate_stacked_spine_frame(frame, boundary="stacked tail preservation") validate_puf_capital_gains_tail_manifest(tail_manifest) + overlap_ownership = tail_manifest.get("late_overlap_ownership") + if not isinstance(overlap_ownership, Mapping): + raise ValueError( + "Stacked tail overlap ownership receipt is absent or malformed." + ) + try: + overlap_ownership_sha256 = validate_us_late_overlap_ownership_receipt( + overlap_ownership + ) + except (TypeError, ValueError) as error: + raise ValueError( + f"Stacked tail overlap ownership receipt is invalid: {error}" + ) from error attachment = validate_puf_clone_attachment( frame, boundary="stacked tail preservation attachment", @@ -5844,6 +10677,7 @@ def assert_float_exact( "tail_owned_state_count": len(observed_state), "recipient_owned_qrf_cell_count": preserved_nonowned, "tail_owned_cells_sha256": _canonical_sha256(observed_state), + "overlap_ownership_sha256": overlap_ownership_sha256, } @@ -5852,9 +10686,67 @@ def assert_float_exact( # --------------------------------------------------------------------------- _COMPLETENESS_GATE_NAME = "us_stacked_completeness" +_TAIL_SUPPORT_GATE_DETAIL_KEY = "puf_capital_gains_tail_support" _ANY_CHANNEL = "*" +def _frame_has_clone_two_rows(frame: Frame) -> bool: + """Return whether any live entity carries the tail-owned clone role.""" + + for entity in frame.entities: + table = frame.table(entity) + clone_column = support_clone_index_column(entity) + if clone_column not in table: + continue + clone_index = pd.to_numeric(table[clone_column], errors="raise") + if clone_index.eq(2).any(): + return True + return False + + +def _terminal_tail_support_gate_receipt( + frame: Frame, + tail_manifest: Mapping[str, object] | None, + *, + boundary: str, +) -> dict[str, object] | None: + """Authenticate the tail support receipt against the live clone identity.""" + + has_clone_two = _frame_has_clone_two_rows(frame) + if tail_manifest is None: + if has_clone_two: + raise ValueError( + f"{boundary}: live clone-2 rows require the bound PUF " + "capital-gains-tail manifest." + ) + return None + if not has_clone_two: + raise ValueError( + f"{boundary}: a supplied PUF capital-gains-tail manifest requires " + "live clone-2 rows." + ) + + validate_puf_capital_gains_tail_manifest(tail_manifest) + terminal_receipt = puf_capital_gains_tail_terminal_support_receipt(tail_manifest) + validate_puf_capital_gains_tail_terminal_support_receipt(terminal_receipt) + + if has_clone_two: + attachment = validate_puf_clone_attachment( + frame, + boundary=f"{boundary} tail attachment", + ) + transform = attachment.get("post_attachment_transform") + if not isinstance(transform, Mapping) or transform.get( + "tail_manifest_sha256" + ) != tail_manifest.get("manifest_sha256"): + raise ValueError( + f"{boundary}: live clone-2 attachment is not bound to the " + "supplied PUF capital-gains-tail manifest." + ) + + return _json_ready(terminal_receipt) + + @dataclass(frozen=True) class AbsenceProof: """An explicit source-by-role authority proof for permitted null cells. @@ -5954,6 +10846,7 @@ def stacked_completeness_gate( frame: Frame, *, absence_proofs: Sequence[AbsenceProof] = (), + tail_manifest: Mapping[str, object] | None = None, ) -> GateResult: """Evaluate the canonical declared surface with no caller authority.""" @@ -5962,6 +10855,7 @@ def stacked_completeness_gate( authority=_production_stacked_authority(), production=True, absence_proofs=absence_proofs, + tail_manifest=tail_manifest, ) @@ -5970,6 +10864,7 @@ def _stacked_completeness_gate_with_test_authority( *, authority: _StackedAuthority, absence_proofs: Sequence[AbsenceProof] = (), + tail_manifest: Mapping[str, object] | None = None, ) -> GateResult: """Explicit test-only completeness seam for a digested authority bundle.""" @@ -5979,6 +10874,7 @@ def _stacked_completeness_gate_with_test_authority( authority=authority, production=False, absence_proofs=absence_proofs, + tail_manifest=tail_manifest, ) @@ -5988,6 +10884,7 @@ def _stacked_completeness_gate_evaluate( authority: _StackedAuthority, production: bool, absence_proofs: Sequence[AbsenceProof], + tail_manifest: Mapping[str, object] | None = None, _canonical_gap_fill_plan: tuple[ GapFillDirection, ... ] = CANONICAL_STACKED_GAP_FILL_PLAN, @@ -6007,6 +10904,11 @@ def _stacked_completeness_gate_evaluate( that target even at the explicitly non-canonical fixture seam. """ + tail_support_receipt = _terminal_tail_support_gate_receipt( + frame, + tail_manifest, + boundary="stacked completeness gate", + ) authority_receipt = _authority_receipt(authority) declared_surface = authority.declared_surface declared_count = len(_surface_target_keys(declared_surface)) @@ -6022,6 +10924,11 @@ def _stacked_completeness_gate_evaluate( "authority": authority_receipt, "declared_targets": declared_count, "targets": {}, + **( + {_TAIL_SUPPORT_GATE_DETAIL_KEY: tail_support_receipt} + if tail_support_receipt is not None + else {} + ), }, ) @@ -6382,6 +11289,11 @@ def authority_binding(authority_form: str) -> dict[str, object]: "authority": authority_receipt, "declared_targets": declared_count, "targets": target_receipts, + **( + {_TAIL_SUPPORT_GATE_DETAIL_KEY: tail_support_receipt} + if tail_support_receipt is not None + else {} + ), }, ) @@ -6453,6 +11365,8 @@ def __post_init__(self) -> None: def by_origin_battery( frame: Frame, + *, + tail_manifest: Mapping[str, object] | None = None, ) -> GateResult: """Run the canonical 131-target plus joint by-origin battery.""" @@ -6460,6 +11374,7 @@ def by_origin_battery( frame, authority=_production_stacked_authority(), production=True, + tail_manifest=tail_manifest, ) @@ -6467,6 +11382,7 @@ def _by_origin_battery_with_test_authority( frame: Frame, *, authority: _StackedAuthority, + tail_manifest: Mapping[str, object] | None = None, ) -> GateResult: """Explicit test-only battery seam for a digested authority bundle.""" @@ -6475,6 +11391,7 @@ def _by_origin_battery_with_test_authority( frame, authority=authority, production=False, + tail_manifest=tail_manifest, ) @@ -6500,6 +11417,7 @@ def _by_origin_battery_evaluate( *, authority: _StackedAuthority, production: bool, + tail_manifest: Mapping[str, object] | None = None, _canonical_gap_fill_plan: tuple[ GapFillDirection, ... ] = CANONICAL_STACKED_GAP_FILL_PLAN, @@ -6523,6 +11441,11 @@ def _by_origin_battery_evaluate( both origins carry ample support. """ + tail_support_receipt = _terminal_tail_support_gate_receipt( + frame, + tail_manifest, + boundary="by-origin battery", + ) authority_receipt = _authority_receipt(authority) specs = _battery_specs_from_metric_registry(authority.metric_registry) registered_targets = set(authority.metric_registry) @@ -6576,6 +11499,11 @@ def _by_origin_battery_evaluate( "tested_comparisons": 0, "untestable_comparisons": [], "comparisons": {}, + **( + {_TAIL_SUPPORT_GATE_DETAIL_KEY: tail_support_receipt} + if tail_support_receipt is not None + else {} + ), }, ) validate_stacked_spine_frame(frame, boundary="by-origin battery") @@ -6832,6 +11760,11 @@ def _by_origin_battery_evaluate( "tested_comparisons": tested, "untestable_comparisons": sorted(untestable), "comparisons": comparisons, + **( + {_TAIL_SUPPORT_GATE_DETAIL_KEY: tail_support_receipt} + if tail_support_receipt is not None + else {} + ), }, ) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_overlap_ownership.py b/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_overlap_ownership.py new file mode 100644 index 00000000..bbcb580f --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_overlap_ownership.py @@ -0,0 +1,262 @@ +"""Final-value ownership for late US targets touched by multiple producers. + +The stacked pipeline intentionally exposes three PUF-recipient targets to a +second post-clone callback before the late ACS transfer pass. This module +turns that historical ordering into a closed, content-addressed owner matrix: +one final owner for every target, origin, and clone role, plus an explicit +disposition for each non-owner. +""" + +from __future__ import annotations + +import hashlib +import json +from collections.abc import Mapping +from types import MappingProxyType + +__all__ = [ + "US_LATE_EDUCATION_NOOP_TARGETS", + "US_LATE_OVERLAP_OWNERSHIP_TARGETS", + "US_LATE_RETIREMENT_SOURCE_MIRROR_TARGETS", + "US_LATE_SOURCE_CALLBACK_PASSTHROUGH_OUTPUTS", + "us_late_overlap_ownership_receipt", + "validate_us_late_overlap_ownership_receipt", +] + +_PRIMARY_PUF_PRODUCER = "primary_puf_qrf" +_EDUCATION_SOURCE_PRODUCER = "source:with_us_education_inputs" +_RETIREMENT_SOURCE_PRODUCER = "source:with_us_retirement_contribution_inputs" +_TUITION_TRANSFER_PRODUCER = "transfer:person/puf_tax_itemization__batch_2" +_TRADITIONAL_IRA_TRANSFER_PRODUCER = "transfer:person/puf_tax_itemization__batch_2" +_SELF_EMPLOYED_PENSION_TRANSFER_PRODUCER = ( + "transfer:person/puf_tax_itemization__batch_3" +) + +_QUALIFIED_TUITION = "qualified_tuition_expenses" +_TRADITIONAL_IRA = "traditional_ira_contributions_desired" +_SELF_EMPLOYED_PENSION = "self_employed_pension_contributions_desired" + +US_LATE_EDUCATION_NOOP_TARGETS = (_QUALIFIED_TUITION,) +US_LATE_RETIREMENT_SOURCE_MIRROR_TARGETS = ( + _TRADITIONAL_IRA, + _SELF_EMPLOYED_PENSION, +) +US_LATE_OVERLAP_OWNERSHIP_TARGETS = tuple( + ("person", target) + for target in ( + _QUALIFIED_TUITION, + _TRADITIONAL_IRA, + _SELF_EMPLOYED_PENSION, + ) +) +US_LATE_SOURCE_CALLBACK_PASSTHROUGH_OUTPUTS: Mapping[ + str, tuple[tuple[str, str], ...] +] = MappingProxyType( + { + "with_us_education_inputs": (("person", _QUALIFIED_TUITION),), + } +) + +_TARGET_SPECS = ( + { + "entity": "person", + "target": _QUALIFIED_TUITION, + "source_producer": _EDUCATION_SOURCE_PRODUCER, + "source_touch": "consume_only_byte_exact_noop", + "transfer_producer": _TUITION_TRANSFER_PRODUCER, + }, + { + "entity": "person", + "target": _TRADITIONAL_IRA, + "source_producer": _RETIREMENT_SOURCE_PRODUCER, + "source_touch": "persisted_owner_last_write", + "transfer_producer": _TRADITIONAL_IRA_TRANSFER_PRODUCER, + }, + { + "entity": "person", + "target": _SELF_EMPLOYED_PENSION, + "source_producer": _RETIREMENT_SOURCE_PRODUCER, + "source_touch": "persisted_owner_last_write", + "transfer_producer": _SELF_EMPLOYED_PENSION_TRANSFER_PRODUCER, + }, +) + + +def _producer_action( + producer: str, + *, + final_owner: str, + action: str, +) -> dict[str, object]: + return { + "producer": producer, + "owns_final": producer == final_owner, + "action": action, + } + + +def _ownership_row( + spec: Mapping[str, str], + *, + origin: str, + clone_index: int, +) -> dict[str, object]: + source = spec["source_producer"] + transfer = spec["transfer_producer"] + tuition = spec["target"] == _QUALIFIED_TUITION + + if tuition: + source_action = ( + "consume_only_byte_exact_noop" + if origin == "asec" + else "origin_projection_masked_noop" + ) + if clone_index == 0: + owner = transfer + finalization = "late_transfer_owner_last" + primary_action = "scope_masked_noop" + transfer_action = "final_write" + else: + owner = _PRIMARY_PUF_PRODUCER + finalization = ( + "primary_write" + if clone_index == 1 + else "byte_exact_clone_1_inheritance" + ) + primary_action = finalization + transfer_action = "producer_masked_byte_exact_noop" + elif origin == "asec": + owner = source + transfer_action = "producer_masked_byte_exact_noop" + if clone_index == 0: + finalization = "source_direct_split" + primary_action = "scope_masked_noop" + source_action = "final_write" + elif clone_index == 1: + finalization = "source_owner_last_overwrite" + primary_action = "interim_write_overwritten_by_owner_last" + source_action = "final_write" + else: + finalization = "byte_exact_clone_1_mirror" + primary_action = "interim_clone_1_inheritance_overwritten_by_owner_last" + source_action = "byte_exact_clone_1_mirror" + elif clone_index == 0: + owner = transfer + finalization = "late_transfer_owner_last" + primary_action = "scope_masked_noop" + source_action = "origin_projection_masked_noop" + transfer_action = "final_write" + else: + owner = _PRIMARY_PUF_PRODUCER + finalization = ( + "primary_write" if clone_index == 1 else "byte_exact_clone_1_inheritance" + ) + primary_action = finalization + source_action = "origin_projection_masked_noop" + transfer_action = "producer_masked_byte_exact_noop" + + actions = [ + _producer_action( + _PRIMARY_PUF_PRODUCER, + final_owner=owner, + action=primary_action, + ), + _producer_action(source, final_owner=owner, action=source_action), + _producer_action(transfer, final_owner=owner, action=transfer_action), + ] + return { + "entity": spec["entity"], + "target": spec["target"], + "origin": origin, + "clone_index": clone_index, + "final_owner": owner, + "finalization": finalization, + "producer_actions": actions, + } + + +def _ownership_payload() -> dict[str, object]: + ownership = [ + _ownership_row(spec, origin=origin, clone_index=clone_index) + for spec in _TARGET_SPECS + for origin in ("asec", "acs") + for clone_index in range(3) + ] + return { + "artifact_kind": "microcosm_us_late_overlap_ownership", + "schema_version": 1, + "doctrine": { + "owner_cardinality": "exactly_one_per_target_origin_clone_role", + "non_owner_write_policy": "masked_or_verified_byte_exact_noop", + "retirement_legacy_order": ( + "primary_then_postclone_source_owner_then_late_transfer" + ), + "clone_2_policy": "inherit_or_mirror_clone_1_final_owner_bytes", + "tail_preservation_guard": "unchanged_fail_closed", + }, + "targets": [dict(spec) for spec in _TARGET_SPECS], + "ownership": ownership, + } + + +def _canonical_sha256(value: object) -> str: + encoded = json.dumps( + value, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + +def us_late_overlap_ownership_receipt() -> Mapping[str, object]: + """Return the canonical content-addressed 3 x 2 x 3 owner matrix.""" + + payload = _ownership_payload() + return MappingProxyType({**payload, "sha256": _canonical_sha256(payload)}) + + +def validate_us_late_overlap_ownership_receipt( + receipt: Mapping[str, object], +) -> str: + """Validate both the digest and exact reviewed overlap-ownership content.""" + + if not isinstance(receipt, Mapping): + raise TypeError("US late overlap ownership receipt must be a mapping.") + observed = dict(receipt) + claimed_sha256 = observed.pop("sha256", None) + if ( + not isinstance(claimed_sha256, str) + or len(claimed_sha256) != 64 + or any(character not in "0123456789abcdef" for character in claimed_sha256) + ): + raise ValueError("US late overlap ownership receipt has an invalid sha256.") + actual_sha256 = _canonical_sha256(observed) + if claimed_sha256 != actual_sha256: + raise ValueError("US late overlap ownership receipt sha256 does not match.") + expected = _ownership_payload() + if observed != expected: + raise ValueError( + "US late overlap ownership receipt differs from the canonical owner matrix." + ) + + ownership = observed["ownership"] + if not isinstance(ownership, list) or len(ownership) != 18: + raise ValueError("US late overlap ownership must contain exactly 18 rows.") + for row in ownership: + if not isinstance(row, Mapping): + raise ValueError("US late overlap ownership rows must be mappings.") + actions = row.get("producer_actions") + if ( + not isinstance(actions, list) + or sum( + action.get("owns_final") is True + for action in actions + if isinstance(action, Mapping) + ) + != 1 + ): + raise ValueError( + "US late overlap ownership requires exactly one final producer." + ) + return claimed_sha256 diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py b/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py new file mode 100644 index 00000000..4a6c9b78 --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py @@ -0,0 +1,2128 @@ +"""Canonical producer-input DAG for the US stacked late stage. + +This module is the data-only production declaration that binds the primary +PUF pass, the sixteen post-clone source operators, and the nineteen bounded +late-transfer groups. It deliberately separates two kinds of input data: + +* :class:`ProducerInput` values are the role-scoped dependencies which can + affect late-stage scheduling; and +* :data:`US_LATE_SOURCE_INPUT_INVENTORIES` records each source kernel's full + effective raw, structural, weight, and optional-input surface. + +Keeping the full inventories beside the executable graph makes an empty +late-dependency set explicit without pretending that a source kernel has no +inputs. The graph itself is validated and topologically sorted at import. +""" + +from __future__ import annotations + +import hashlib +import json +from collections import defaultdict +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from types import MappingProxyType + +from microcosm.build.us_runtime.acs_income_universe import ( + ACS_PUMS_EARNINGS_SOURCE_COLUMNS, +) +from microcosm.build.us_runtime.acs_transfer import TargetFamilies +from microcosm.build.us_runtime.education_inputs import ( + US_EDUCATION_INPUTS_OUTPUT_COLUMNS, + US_EDUCATION_INPUTS_OWNED_OUTPUT_COLUMNS, +) +from microcosm.build.us_runtime.late_producer_dag import ( + ProducerContract, + ProducerInput, + ProducerInputColumn, + ProducerOutput, + ProducerSchedule, + derive_producer_schedule, +) +from microcosm.build.us_runtime.multispine_pool import ( + POOL_DEFERRED_TRANSFER_INPUTS, + POOL_OPERATOR_CONTRACTS, + POOL_POST_CLONE_SOURCE_OPERATOR_ORDER, + pool_post_puf_puf_producer_target_families, + pool_post_puf_transfer_target_families, +) +from microcosm.build.us_runtime.operator_boundary import ( + FORMULA_OWNED_SOURCE_COLUMNS, + PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES, +) +from microcosm.build.us_runtime.puf_capital_gains_tail import ( + PUF_CAPITAL_GAINS_TAIL_PERSON_COLUMNS, + PUF_CAPITAL_GAINS_TAIL_TAX_UNIT_COLUMNS, +) +from microcosm.build.us_runtime.puf_support import ( + PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS, + PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS, +) +from microcosm.build.us_runtime.us_late_overlap_ownership import ( + US_LATE_OVERLAP_OWNERSHIP_TARGETS, + US_LATE_SOURCE_CALLBACK_PASSTHROUGH_OUTPUTS, + us_late_overlap_ownership_receipt, +) + +__all__ = [ + "CANONICAL_US_LATE_PRODUCER_REGISTRY", + "CANONICAL_US_LATE_PRODUCER_SCHEDULE", + "CANONICAL_US_LATE_SOURCE_OUTPUTS", + "CANONICAL_US_LATE_TRANSFER_GROUPS", + "EffectiveInputRequirement", + "ScopedInput", + "SourceInputInventory", + "TransferProducerGroup", + "US_LATE_EXTERNAL_STAGES", + "US_LATE_ACS_EARNINGS_UNIVERSE_CONFIG_INPUT", + "US_LATE_ACS_EARNINGS_UNIVERSE_INPUT_INVENTORY", + "US_LATE_ACS_EARNINGS_UNIVERSE_RECEIPT_INPUT", + "US_LATE_ACS_EARNINGS_UNIVERSE_STAGE", + "US_LATE_PRIMARY_EXECUTION_CONFIG_INPUT", + "US_LATE_PRIMARY_PUF_STAGE", + "US_LATE_SOURCE_FINALIZER_STAGE", + "US_LATE_SOURCE_EXECUTION_CONFIG_INPUT", + "US_LATE_SOURCE_FINALIZER_EXECUTION_CONFIG_INPUT", + "US_LATE_PRIMARY_PUF_INPUT_INVENTORY", + "US_LATE_PRODUCER_RECEIPT_SCHEMA_VERSION", + "US_LATE_PRODUCER_REGISTRY_SCHEMA_VERSION", + "US_LATE_PRODUCER_TRANSITION_AUTHORITY_ID", + "US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY", + "US_LATE_PRODUCER_TRANSITION_AUTHORITY_VERSION", + "US_LATE_SOURCE_INPUT_INVENTORIES", + "US_LATE_TRANSFER_INPUT_INVENTORIES", + "US_LATE_TRANSFER_MODEL_CONFIG_INPUT", + "US_LATE_TRANSFER_TARGET_BANK_INPUT", + "source_producer_name", + "transfer_producer_name", + "us_late_producer_schedule_payload", + "us_late_producer_schedule_receipt", +] + +# v16 declares person.s_corp_income as a whole-pool primary-PUF output: its +# certified combined-source semantics are carried by partnership_income, while +# the separate S-corporation leaf is an exact-zero universe. v15 content-binds +# the complete late dual-producer ownership matrix and +# validates that it exhausts the primary/source/transfer intersection. v14 +# scopes origin-exclusive raw requirements independently of their inventory +# defaults and retires whole-pool RELSHIPP/TEN/H_TENURE transfer fallbacks. v13 +# declares the primary callback's optional tax-unit pass-through reads and +# binds its complete tail-control/runtime-asset surface. v12 declared every +# primary callback person read-before-write and universe-validation column and +# removed the unusable filing-status fallback. v11 bound the complete +# packaged SourceStageSpec/default surface of every +# source callback and the source finalizer's registry/exclusion/deferral +# doctrine. v10 completed the ACS PUMS earnings-universe input declaration with +# its tax-unit link, clone role, and stable lineage fallback. v9 split that +# materializer into a declared pre-primary producer. v8 added the fixed +# seed/period and operator switches consumed by every post-clone source callback. +# v7 added the primary execution configuration and every late-transfer model +# configuration/target-bank identity to the declared external-resource surface. +# Version 6 content-bound physical Frame inputs but left those callback inputs +# implicit. Receipt v3 reconciles repeated physical evidence and scope +# cardinalities across each execution row, binds source-receipt outputs to the +# callback receipt, and requires the primary callback to report the exact +# resources it consumed. Receipt v2 introduced exact virtual-resource payloads. +US_LATE_PRODUCER_REGISTRY_SCHEMA_VERSION = 16 +US_LATE_PRODUCER_RECEIPT_SCHEMA_VERSION = 3 +US_LATE_PRODUCER_TRANSITION_AUTHORITY_VERSION = 1 +US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY = "us_late_producer_transition_authority" +US_LATE_PRODUCER_TRANSITION_AUTHORITY_ID = "us_stacked_late_producer_transition" +US_LATE_PRIMARY_PUF_STAGE = "primary_puf_qrf" +US_LATE_ACS_EARNINGS_UNIVERSE_STAGE = "acs_pums_earnings_universe" +US_LATE_SOURCE_FINALIZER_STAGE = "source_finalizer" +US_LATE_ACS_EARNINGS_UNIVERSE_CONFIG_INPUT = ( + "@acs_pums_earnings_universe_execution_config" +) +US_LATE_ACS_EARNINGS_UNIVERSE_RECEIPT_INPUT = "@acs_pums_earnings_universe_application" +US_LATE_SOURCE_EXECUTION_CONFIG_INPUT = "@post_clone_source_execution_config" +US_LATE_SOURCE_FINALIZER_EXECUTION_CONFIG_INPUT = "@source_finalizer_execution_config" +US_LATE_EXTERNAL_STAGES: tuple[str, ...] = ("post_clone_input_surface",) + +_ASEC_SOURCE_SCOPE = "asec_source" +_ACS_SOURCE_SCOPE = "acs_source" +_PUF_CLONE_SCOPE = "puf_clone" +_WHOLE_POOL_SCOPE = "whole_pool" +_DEFAULT_MAX_TARGETS_PER_FIT = 8 +_QUALIFIED_TUITION = "qualified_tuition_expenses" +_SSTB_EARNED_INCOME = "sstb_self_employment_income_before_lsr" +_CHILDCARE_OUTPUT = "spm_unit_pre_subsidy_childcare_expenses" +_PREGNANCY_OUTPUT = "is_pregnant" +_ADULT_CARE_ROLE_INPUT = "tax_unit_role_input" +_CLONE_ATTACHMENT_OUTPUT = "person_support_clone_index" +_SOURCE_RECEIPT_PREFIX = "@source_receipt:" +US_LATE_PRIMARY_EXECUTION_CONFIG_INPUT = "@primary_puf_execution_config" +US_LATE_TRANSFER_MODEL_CONFIG_INPUT = "@late_transfer_model_config" +US_LATE_TRANSFER_TARGET_BANK_INPUT = "@late_transfer_target_bank" +_STRUCTURAL_ENTITIES = ( + "person", + "household", + "tax_unit", + "spm_unit", + "family", + "marital_unit", +) +_GROUP_ENTITIES = _STRUCTURAL_ENTITIES[1:] +_ASSEMBLY_MANIFEST_INPUT = "@us_spine_assembly_manifest" +_STACKED_MANIFEST_INPUT = "@us_stacked_spine_manifest" +_PUF_ATTACHMENT_MANIFEST_INPUT = "@us_puf_clone_attachment_manifest" +_STRING_LATE_TARGETS = frozenset({"ssn_card_type", "immigration_status_str"}) +_BOOLEAN_LATE_TARGETS = frozenset( + { + "is_incapable_of_self_care", + "is_pregnant", + "estate_income_would_be_qualified", + "farm_operations_income_would_be_qualified", + "farm_rent_income_would_be_qualified", + "partnership_s_corp_income_would_be_qualified", + "rental_income_would_be_qualified", + "self_employment_income_would_be_qualified", + "sstb_self_employment_income_would_be_qualified", + "business_is_sstb", + "attends_eligible_educational_institution_for_american_opportunity_credit", + "has_american_opportunity_credit_1098_t_or_exception", + "has_american_opportunity_credit_institution_ein", + "is_enrolled_at_least_half_time_for_american_opportunity_credit", + "is_pursuing_credential_for_american_opportunity_credit", + "takes_up_medicare_if_eligible", + "would_claim_wic", + } +) + + +def _nonempty(value: object, *, label: str) -> str: + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"{label} must be a non-empty string.") + return value + + +def _late_target_value_kind(column: str) -> str: + if column in _BOOLEAN_LATE_TARGETS or column in _STRING_LATE_TARGETS: + return "non_null" + return "finite_numeric" + + +ScopedInput = ProducerInputColumn + + +@dataclass(frozen=True) +class EffectiveInputRequirement: + """One logical source input, expressed as one or more all-of alternatives.""" + + label: str + alternatives: tuple[tuple[ScopedInput, ...], ...] + optional: bool = False + required_scope: str | None = None + + def __post_init__(self) -> None: + _nonempty(self.label, label="EffectiveInputRequirement.label") + alternatives = tuple(tuple(option) for option in self.alternatives) + if not alternatives or any(not option for option in alternatives): + raise ValueError( + f"Effective input {self.label!r} requires nonempty alternatives." + ) + if any( + not isinstance(item, ScopedInput) + for option in alternatives + for item in option + ): + raise TypeError( + f"Effective input {self.label!r} alternatives require " + "ScopedInput values." + ) + canonical = tuple( + sorted( + (tuple(sorted(set(option))) for option in alternatives), + key=lambda option: tuple( + (item.entity, item.column, item.value_kind) for item in option + ), + ) + ) + if len(set(canonical)) != len(canonical): + raise ValueError(f"Effective input {self.label!r} repeats an alternative.") + object.__setattr__(self, "alternatives", canonical) + if self.required_scope is not None: + _nonempty( + self.required_scope, + label=f"Effective input {self.label!r} required_scope", + ) + + +@dataclass(frozen=True) +class SourceInputInventory: + """Complete effective input declaration for one post-clone source kernel.""" + + operator: str + requirements: tuple[EffectiveInputRequirement, ...] + + def __post_init__(self) -> None: + _nonempty(self.operator, label="SourceInputInventory.operator") + requirements = tuple(self.requirements) + if not requirements: + raise ValueError( + f"Source input inventory {self.operator!r} must not be empty." + ) + if any( + not isinstance(item, EffectiveInputRequirement) for item in requirements + ): + raise TypeError( + f"Source input inventory {self.operator!r} requires " + "EffectiveInputRequirement values." + ) + labels = [item.label for item in requirements] + if len(set(labels)) != len(labels): + raise ValueError( + f"Source input inventory {self.operator!r} repeats labels: {labels}." + ) + object.__setattr__( + self, + "requirements", + tuple(sorted(requirements, key=lambda item: item.label)), + ) + + +@dataclass(frozen=True) +class TransferProducerGroup: + """One canonical bounded ACS-transfer family represented by a DAG node.""" + + name: str + entity: str + family: str + targets: tuple[str, ...] + target_families: TargetFamilies + + def __post_init__(self) -> None: + _nonempty(self.name, label="TransferProducerGroup.name") + _nonempty(self.entity, label="TransferProducerGroup.entity") + _nonempty(self.family, label="TransferProducerGroup.family") + targets = tuple(self.targets) + if not targets or any( + not isinstance(item, str) or not item for item in targets + ): + raise ValueError( + f"Transfer producer group {self.name!r} requires named targets." + ) + if len(set(targets)) != len(targets): + raise ValueError(f"Transfer producer group {self.name!r} repeats targets.") + expected = {self.entity: {self.family: targets}} + materialized = { + entity: {family: tuple(columns) for family, columns in families.items()} + for entity, families in self.target_families.items() + } + if materialized != expected: + raise ValueError( + f"Transfer producer group {self.name!r} target_families drifted; " + f"expected {expected}, got {materialized}." + ) + object.__setattr__(self, "targets", targets) + + +def source_producer_name(operator: str) -> str: + """Return the graph node name for one post-clone source operator.""" + + return f"source:{_nonempty(operator, label='source operator')}" + + +def transfer_producer_name(entity: str, family: str) -> str: + """Return the graph node name for one bounded late-transfer family.""" + + return ( + f"transfer:{_nonempty(entity, label='transfer entity')}/" + f"{_nonempty(family, label='transfer family')}" + ) + + +def _column( + entity: str, + column: str, + *, + value_kind: str = "non_null", +) -> ScopedInput: + return ScopedInput(entity, column, value_kind) + + +def _requirement( + label: str, + *alternatives: Sequence[ScopedInput], + optional: bool = False, + required_scope: str | None = None, +) -> EffectiveInputRequirement: + return EffectiveInputRequirement( + label=label, + alternatives=tuple(tuple(option) for option in alternatives), + optional=optional, + required_scope=required_scope, + ) + + +def _single( + label: str, + entity: str, + column: str, + *, + optional: bool = False, + value_kind: str = "non_null", + required_scope: str | None = None, +) -> EffectiveInputRequirement: + return _requirement( + label, + (_column(entity, column, value_kind=value_kind),), + optional=optional, + required_scope=required_scope, + ) + + +def _cross_grain_validation_requirements() -> tuple[EffectiveInputRequirement, ...]: + """Return the physical and metadata surface read by stacked validators.""" + + requirements: list[EffectiveInputRequirement] = [] + for entity in _STRUCTURAL_ENTITIES: + requirements.extend( + ( + _single( + f"validated_structure:{entity}_support_channel", + entity, + f"{entity}_support_channel", + ), + _single( + f"validated_structure:{entity}_support_clone_index", + entity, + f"{entity}_support_clone_index", + value_kind="finite_numeric", + ), + ) + ) + for entity in _GROUP_ENTITIES: + requirements.extend( + ( + _single( + f"validated_structure:person_{entity}_id", + "person", + f"person_{entity}_id", + value_kind="finite_numeric", + ), + _single( + f"validated_structure:{entity}_id", + entity, + f"{entity}_id", + value_kind="finite_numeric", + ), + ) + ) + requirements.extend( + ( + _single( + "validated_structure:person_id", + "person", + "person_id", + value_kind="finite_numeric", + ), + _single( + "validated_structure:person_spine_source_id", + "person", + "person_spine_source_id", + value_kind="finite_numeric", + ), + _single( + "validated_structure:person_source_id", + "person", + "person_source_id", + value_kind="finite_numeric", + ), + _single( + "validated_structure:household_spine_source_id", + "household", + "household_spine_source_id", + value_kind="finite_numeric", + ), + _single( + "validated_structure:household_source_id", + "household", + "household_source_id", + value_kind="finite_numeric", + ), + _single( + "validated_structure:TYPEHUGQ", + "household", + "TYPEHUGQ", + value_kind="finite_numeric", + required_scope=_ACS_SOURCE_SCOPE, + ), + _single( + "validated_structure:resolved_household_weight", + "household", + "@resolved_weight", + ), + _single( + "validated_structure:assembly_manifest", + "frame", + _ASSEMBLY_MANIFEST_INPUT, + ), + _single( + "validated_structure:stacked_manifest", + "frame", + _STACKED_MANIFEST_INPUT, + ), + _single( + "validated_structure:puf_attachment_manifest", + "frame", + _PUF_ATTACHMENT_MANIFEST_INPUT, + ), + ) + ) + return tuple(requirements) + + +_CROSS_GRAIN_VALIDATION_REQUIREMENTS = _cross_grain_validation_requirements() + + +_POST_CLONE_SOURCE_WRAPPER_REQUIREMENTS = ( + _single( + "source_wrapper:execution_config", + "person", + US_LATE_SOURCE_EXECUTION_CONFIG_INPUT, + ), + _single( + "source_wrapper:assembly_manifest", + "frame", + _ASSEMBLY_MANIFEST_INPUT, + ), + _single( + "source_wrapper:source_evidence", + "person", + "PERIDNUM", + ), + _single( + "source_wrapper:person_support_clone_index", + "person", + "person_support_clone_index", + value_kind="finite_numeric", + ), + *tuple( + _single( + f"source_wrapper:{entity}_id", + entity, + f"{entity}_id", + value_kind="finite_numeric", + ) + for entity in _STRUCTURAL_ENTITIES + ), + *tuple( + _single( + f"source_wrapper:person_{entity}_id", + "person", + f"person_{entity}_id", + value_kind="finite_numeric", + ) + for entity in _GROUP_ENTITIES + ), + _single( + "source_wrapper:resolved_household_weight", + "household", + "@resolved_weight", + ), +) + + +_COMMON_ROLE_AWARE_INPUTS = ( + _single( + "person_id", + "person", + "person_id", + value_kind="finite_numeric", + ), + _single("resolved_person_weight", "person", "@resolved_weight"), + _single("support_channel", "person", "person_support_channel"), + _single( + "support_clone_index", + "person", + "person_support_clone_index", + optional=True, + value_kind="finite_numeric", + ), + _requirement( + "age", + (_column("person", "age", value_kind="finite_numeric"),), + (_column("person", "A_AGE", value_kind="finite_numeric"),), + ), + _requirement( + "sex", + (_column("person", "is_male", value_kind="finite_numeric"),), + (_column("person", "is_female", value_kind="finite_numeric"),), + (_column("person", "A_SEX", value_kind="finite_numeric"),), + ), + _single( + "employer_health_coverage", + "person", + "has_esi", + value_kind="finite_numeric", + ), + _single( + "person_tax_unit_link", + "person", + "person_tax_unit_id", + value_kind="finite_numeric", + ), + _single("tax_unit_role", "person", "tax_unit_role_input"), + _requirement( + "employment_income", + ( + _column( + "person", + "employment_income_before_lsr", + value_kind="finite_numeric", + ), + ), + (_column("person", "WSAL_VAL", value_kind="finite_numeric"),), + ), + _requirement( + "self_employment_income", + ( + _column( + "person", + "self_employment_income_before_lsr", + value_kind="finite_numeric", + ), + ), + (_column("person", "SEMP_VAL", value_kind="finite_numeric"),), + ), + _requirement( + "social_security_income", + ( + _column( + "person", "social_security_retirement", value_kind="finite_numeric" + ), + _column( + "person", "social_security_disability", value_kind="finite_numeric" + ), + _column("person", "social_security_survivors", value_kind="finite_numeric"), + _column( + "person", "social_security_dependents", value_kind="finite_numeric" + ), + ), + (_column("person", "SS_VAL", value_kind="finite_numeric"),), + ), + _single( + "tax_unit_id", + "tax_unit", + "tax_unit_id", + value_kind="finite_numeric", + ), + _requirement( + "filing_status", + (_column("tax_unit", "filing_status_input"),), + (_column("tax_unit", "filing_status"),), + ), +) + + +def _raw_person_requirements( + columns: Sequence[str], +) -> tuple[EffectiveInputRequirement, ...]: + return tuple( + _single( + f"raw_person:{column}", + "person", + column, + value_kind="finite_numeric", + ) + for column in columns + ) + + +def _inventory( + operator: str, + *requirements: EffectiveInputRequirement, +) -> SourceInputInventory: + return SourceInputInventory(operator, tuple(requirements)) + + +# These inventories spell out what the wrappers read, including alternate +# canonical/raw spellings and inputs which are optional only because a pinned +# sidecar or stable identity fallback exists. ``@resolved_weight`` denotes a +# typed Frame weight, not a physical table column. +_source_input_inventories = { + "with_us_prior_year_income_inputs": _inventory( + "with_us_prior_year_income_inputs", + *_raw_person_requirements( + ("source_year", "PERIDNUM", "WSAL_VAL", "SEMP_VAL", "I_ERNVAL", "I_SEVAL") + ), + *_COMMON_ROLE_AWARE_INPUTS, + _single( + "employment_income_last_year", + "person", + "employment_income_last_year", + value_kind="finite_numeric", + ), + _single( + "self_employment_income_last_year", + "person", + "self_employment_income_last_year", + value_kind="finite_numeric", + ), + ), + "with_us_medicare_take_up_input": _inventory( + "with_us_medicare_take_up_input", + *_raw_person_requirements(("MCARE",)), + _single("person_id", "person", "person_id"), + _single("resolved_person_weight", "person", "@resolved_weight"), + ), + "with_us_pregnancy_inputs": _inventory( + "with_us_pregnancy_inputs", + *_raw_person_requirements(("A_SEX", "A_AGE")), + _single("person_id", "person", "person_id"), + _single("resolved_person_weight", "person", "@resolved_weight"), + _requirement( + "stable_source_identity", + ( + _column("person", "source_year"), + _column("person", "source_household_id"), + _column("person", "source_person_id"), + ), + (_column("person", "person_id"),), + ), + ), + "with_us_wic_claim_input": _inventory( + "with_us_wic_claim_input", + *_raw_person_requirements( + ( + "age", + "is_female", + _PREGNANCY_OUTPUT, + "own_children_in_household", + "person_family_id", + ) + ), + _single("resolved_person_weight", "person", "@resolved_weight"), + _requirement( + "stable_source_identity", + ( + _column("person", "source_year"), + _column("person", "source_household_id"), + _column("person", "source_person_id"), + ), + (_column("person", "person_support_source_id"),), + (_column("person", "person_id"),), + ), + ), + "impute_us_housing_assistance_to_puf_support": _inventory( + "impute_us_housing_assistance_to_puf_support", + *_COMMON_ROLE_AWARE_INPUTS, + _single("person_spm_unit_link", "person", "person_spm_unit_id"), + _single("spm_unit_id", "spm_unit", "spm_unit_id"), + _single( + "housing_assistance_receipt", + "spm_unit", + "receives_housing_assistance", + value_kind="finite_numeric", + ), + _single( + "housing_assistance_takeup", + "spm_unit", + "takes_up_housing_assistance_if_eligible", + value_kind="finite_numeric", + ), + _single("spm_support_channel", "spm_unit", "spm_unit_support_channel"), + _single( + "spm_support_clone_index", + "spm_unit", + "spm_unit_support_clone_index", + optional=True, + ), + ), + "with_us_child_support_inputs": _inventory( + "with_us_child_support_inputs", + *_raw_person_requirements(("CSP_VAL", "CHSP_VAL")), + *_COMMON_ROLE_AWARE_INPUTS, + ), + "with_us_disability_benefits": _inventory( + "with_us_disability_benefits", + *_raw_person_requirements(("DIS_VAL1", "DIS_SC1", "DIS_VAL2", "DIS_SC2")), + *_COMMON_ROLE_AWARE_INPUTS, + ), + "with_us_workers_compensation": _inventory( + "with_us_workers_compensation", + *_raw_person_requirements(("WC_VAL",)), + *_COMMON_ROLE_AWARE_INPUTS, + ), + "with_us_weeks_unemployed": _inventory( + "with_us_weeks_unemployed", + _single( + "source_year", + "person", + "source_year", + value_kind="finite_numeric", + ), + _single( + "source_identity", + "person", + "PERIDNUM", + value_kind="finite_numeric", + ), + _requirement( + "weeks_source", + (_column("person", "LKWEEKS", value_kind="finite_numeric"),), + ), + _requirement( + "age", + (_column("person", "age", value_kind="finite_numeric"),), + (_column("person", "A_AGE", value_kind="finite_numeric"),), + ), + _requirement( + "sex", + (_column("person", "is_male", value_kind="finite_numeric"),), + (_column("person", "is_female", value_kind="finite_numeric"),), + (_column("person", "A_SEX", value_kind="finite_numeric"),), + ), + _requirement( + "joint_filing_status", + (_column("person", "tax_unit_is_joint", value_kind="finite_numeric"),), + ( + _column("person", "person_tax_unit_id"), + _column("tax_unit", "tax_unit_id"), + _column("tax_unit", "filing_status_input"), + ), + ( + _column("person", "person_tax_unit_id"), + _column("tax_unit", "tax_unit_id"), + _column("tax_unit", "filing_status"), + ), + ), + _requirement( + "explicit_tax_unit_roles", + (_column("person", "tax_unit_role_input"),), + ( + _column("person", "is_tax_unit_head", value_kind="finite_numeric"), + _column("person", "is_tax_unit_spouse", value_kind="finite_numeric"), + _column("person", "is_tax_unit_dependent", value_kind="finite_numeric"), + ), + ), + _requirement( + "unemployment_compensation_predictor", + ( + _column( + "person", + "unemployment_compensation", + value_kind="finite_numeric", + ), + ), + (_column("person", "UC_VAL", value_kind="finite_numeric"),), + optional=True, + ), + _single("support_channel", "person", "person_support_channel"), + _single("resolved_person_weight", "person", "@resolved_weight"), + ), + "with_us_childcare_inputs": _inventory( + "with_us_childcare_inputs", + *_raw_person_requirements(("person_spm_unit_id", "SPM_CHILDCAREXPNS")), + *_COMMON_ROLE_AWARE_INPUTS, + _single("spm_unit_id", "spm_unit", "spm_unit_id"), + ), + "with_us_adult_care_inputs": _inventory( + "with_us_adult_care_inputs", + _single( + "raw_person:PEDISDRS", + "person", + "PEDISDRS", + value_kind="finite_numeric", + ), + _single( + "raw_person:is_full_time_college_student", + "person", + "is_full_time_college_student", + value_kind="finite_numeric", + ), + _single("age", "person", "age", value_kind="finite_numeric"), + _single( + "employment_income", + "person", + "employment_income_before_lsr", + value_kind="finite_numeric", + ), + _single( + "self_employment_income", + "person", + "self_employment_income_before_lsr", + value_kind="finite_numeric", + ), + _single( + "sstb_earned_income", + "person", + _SSTB_EARNED_INCOME, + value_kind="finite_numeric", + ), + _single("tax_unit_role", "person", "tax_unit_role_input"), + _single( + "person_tax_unit_link", + "person", + "person_tax_unit_id", + value_kind="finite_numeric", + ), + _single( + "person_spm_unit_link", + "person", + "person_spm_unit_id", + value_kind="finite_numeric", + ), + _single( + "person_id", + "person", + "person_id", + value_kind="finite_numeric", + ), + _requirement( + "support_role", + ( + _column( + "person", + "person_support_clone_index", + value_kind="finite_numeric", + ), + _column("person", "person_support_channel"), + ), + ), + _single( + "childcare_expenses", + "spm_unit", + _CHILDCARE_OUTPUT, + value_kind="finite_numeric", + ), + _single( + "spm_unit_id", + "spm_unit", + "spm_unit_id", + value_kind="finite_numeric", + ), + _single( + "tax_unit_id", + "tax_unit", + "tax_unit_id", + value_kind="finite_numeric", + ), + _single("resolved_person_weight", "person", "@resolved_weight"), + _single("resolved_tax_unit_weight", "tax_unit", "@resolved_weight"), + _single("resolved_spm_unit_weight", "spm_unit", "@resolved_weight"), + ), + "with_us_energy_subsidy_input": _inventory( + "with_us_energy_subsidy_input", + *_raw_person_requirements(("person_spm_unit_id", "SPM_ENGVAL")), + *_COMMON_ROLE_AWARE_INPUTS, + _single("spm_unit_id", "spm_unit", "spm_unit_id"), + ), + "with_us_retirement_contribution_inputs": _inventory( + "with_us_retirement_contribution_inputs", + *_raw_person_requirements(("RETCB_VAL", "WSAL_VAL", "SEMP_VAL")), + *_COMMON_ROLE_AWARE_INPUTS, + ), + "with_us_retirement_distribution_inputs": _inventory( + "with_us_retirement_distribution_inputs", + *_raw_person_requirements( + ( + "DST_SC1", + "DST_VAL1", + "DST_SC2", + "DST_VAL2", + "DST_SC1_YNG", + "DST_VAL1_YNG", + "DST_SC2_YNG", + "DST_VAL2_YNG", + ) + ), + *_COMMON_ROLE_AWARE_INPUTS, + _single( + "puf_taxable_ira_distribution", + "person", + "taxable_ira_distributions", + value_kind="finite_numeric", + ), + ), + "with_us_immigration_inputs": _inventory( + "with_us_immigration_inputs", + *_raw_person_requirements( + ( + "PRCITSHP", + "PEINUSYR", + "PENATVTY", + "A_AGE", + "A_MARITL", + "A_SPOUSE", + "A_HSCOL", + "WSAL_VAL", + "SEMP_VAL", + "MCARE", + "CAID", + "IHSFLG", + "CHAMPVA", + "MIL", + "PEN_SC1", + "PEN_SC2", + "RESNSS1", + "RESNSS2", + "SS_YN", + "SSI_YN", + "PEIO1COW", + "A_MJOCC", + "PEAFEVER", + "SPM_CAPHOUSESUB", + ) + ), + _single("person_id", "person", "person_id"), + _single("resolved_person_weight", "person", "@resolved_weight"), + _requirement( + "stable_source_identity", + ( + _column("person", "source_year"), + _column("person", "source_person_id"), + ), + (_column("person", "person_id"),), + ), + ), + "with_us_education_inputs": _inventory( + "with_us_education_inputs", + _requirement( + "education_source", + (_column("person", "ED_VAL", value_kind="finite_numeric"),), + ), + _single( + "qualified_tuition", + "person", + _QUALIFIED_TUITION, + value_kind="finite_numeric", + ), + _single("person_id", "person", "person_id"), + _single("resolved_person_weight", "person", "@resolved_weight"), + ), +} + +_source_input_inventories = { + operator: SourceInputInventory( + operator, + (*inventory.requirements, *_POST_CLONE_SOURCE_WRAPPER_REQUIREMENTS), + ) + for operator, inventory in _source_input_inventories.items() +} + +if set(_source_input_inventories) != set(POOL_POST_CLONE_SOURCE_OPERATOR_ORDER): + raise RuntimeError( + "US late source input inventories must cover the exact post-clone " + "operator registry; " + f"missing={sorted(set(POOL_POST_CLONE_SOURCE_OPERATOR_ORDER) - set(_source_input_inventories))}, " + f"extra={sorted(set(_source_input_inventories) - set(POOL_POST_CLONE_SOURCE_OPERATOR_ORDER))}." + ) +US_LATE_SOURCE_INPUT_INVENTORIES: Mapping[str, SourceInputInventory] = MappingProxyType( + dict(_source_input_inventories) +) + + +US_LATE_PRIMARY_PUF_INPUT_INVENTORY = _inventory( + US_LATE_PRIMARY_PUF_STAGE, + *( + requirement + for requirement in _CROSS_GRAIN_VALIDATION_REQUIREMENTS + if requirement.label != "validated_structure:puf_attachment_manifest" + ), + _single("filing_status", "tax_unit", "filing_status_input"), + _single("age", "person", "age", value_kind="finite_numeric"), + _requirement( + "tax_unit_person_count", + ( + _column("person", "person_tax_unit_id"), + _column("tax_unit", "tax_unit_id"), + ), + ), + _single( + "employment_income", + "person", + "employment_income_before_lsr", + value_kind="finite_numeric", + ), + _single( + "self_employment_income", + "person", + "self_employment_income_before_lsr", + value_kind="finite_numeric", + ), + _single( + "taxable_interest_income", + "person", + "taxable_interest_income", + value_kind="finite_numeric", + ), + _requirement( + "dividend_income", + (_column("person", "dividend_income", value_kind="finite_numeric"),), + ( + _column("person", "qualified_dividend_income", value_kind="finite_numeric"), + _column( + "person", + "non_qualified_dividend_income", + value_kind="finite_numeric", + ), + ), + (_column("tax_unit", "dividend_income", value_kind="finite_numeric"),), + ), + _requirement( + "short_term_capital_gains", + (_column("person", "short_term_capital_gains", value_kind="finite_numeric"),), + (_column("tax_unit", "short_term_capital_gains", value_kind="finite_numeric"),), + ), + _requirement( + "long_term_capital_gains", + ( + _column( + "person", + "long_term_capital_gains_before_response", + value_kind="finite_numeric", + ), + ), + (_column("person", "long_term_capital_gains", value_kind="finite_numeric"),), + (_column("tax_unit", "long_term_capital_gains", value_kind="finite_numeric"),), + ), + _single("person_id", "person", "person_id"), + _single("tax_unit_id", "tax_unit", "tax_unit_id"), + _single("support_channel", "person", "person_support_channel"), + _single("support_clone_index", "person", "person_support_clone_index"), + _single("resolved_tax_unit_weight", "tax_unit", "@resolved_weight"), + *( + _single( + f"person_output_allocation_basis:{column}", + "person", + column, + optional=True, + value_kind="finite_numeric", + ) + for column in PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS + ), + *( + _single( + f"tax_unit_output_passthrough:{column}", + "tax_unit", + column, + optional=True, + value_kind="finite_numeric", + ) + for column in PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS + ), + _single( + "qualified_tuition_allocation_fallback", + "person", + "is_full_time_college_student", + optional=True, + value_kind="finite_numeric", + ), + _single("puf_donor", "tax_unit", "@puf_donor_tax_units"), + _single("primary_qrf_bank", "tax_unit", "@primary_qrf_checkpoint"), + _single( + "primary_puf_execution_config", + "tax_unit", + US_LATE_PRIMARY_EXECUTION_CONFIG_INPUT, + ), +) + + +US_LATE_ACS_EARNINGS_UNIVERSE_INPUT_INVENTORY = _inventory( + US_LATE_ACS_EARNINGS_UNIVERSE_STAGE, + _single("age", "person", "age", value_kind="finite_numeric"), + _single("support_channel", "person", "person_support_channel"), + _single("person_tax_unit_link", "person", "person_tax_unit_id"), + _single( + "support_clone_index", + "person", + "person_support_clone_index", + value_kind="finite_numeric", + ), + _requirement( + "stable_person_lineage", + (_column("person", "person_source_id"),), + (_column("person", "person_id"),), + ), + *( + _single( + f"raw_source:{source}", + "person", + source, + value_kind="column_present", + ) + for source in ACS_PUMS_EARNINGS_SOURCE_COLUMNS.values() + ), + *( + _single( + f"mapped_earnings:{mapped}", + "person", + mapped, + optional=True, + value_kind="finite_numeric", + ) + for mapped in ACS_PUMS_EARNINGS_SOURCE_COLUMNS + ), + _single( + "execution_config", + "person", + US_LATE_ACS_EARNINGS_UNIVERSE_CONFIG_INPUT, + ), +) + + +def _transfer_input_inventory(group: TransferProducerGroup) -> SourceInputInventory: + structural = [ + *_CROSS_GRAIN_VALIDATION_REQUIREMENTS, + _single("resolved_person_weight", "person", "@resolved_weight"), + _single("resolved_target_weight", group.entity, "@resolved_weight"), + _single( + "late_transfer_model_config", + group.entity, + US_LATE_TRANSFER_MODEL_CONFIG_INPUT, + ), + _single( + "late_transfer_target_bank", + group.entity, + US_LATE_TRANSFER_TARGET_BANK_INPUT, + ), + ] + post_transfer_structure: list[EffectiveInputRequirement] = [] + if group.name == transfer_producer_name("person", "adult_care"): + post_transfer_structure.append( + _single( + "adult_care_tax_unit_role", + "person", + _ADULT_CARE_ROLE_INPUT, + ) + ) + return _inventory( + group.name, + *structural, + *post_transfer_structure, + _single("age", "person", "age", value_kind="finite_numeric"), + _single( + "is_female", + "person", + "is_female", + value_kind="finite_numeric", + ), + _requirement( + "state_fips", + (_column("person", "state_fips", value_kind="finite_numeric"),), + ( + _column("person", "person_household_id", value_kind="finite_numeric"), + _column("household", "household_id", value_kind="finite_numeric"), + _column("household", "state_fips", value_kind="finite_numeric"), + ), + ), + _single( + "optional_employment_income", + "person", + "employment_income_before_lsr", + optional=True, + value_kind="finite_numeric", + ), + _single( + "optional_self_employment_income", + "person", + "self_employment_income_before_lsr", + optional=True, + value_kind="finite_numeric", + ), + _requirement( + "optional_social_security_income", + ( + _column( + "person", "social_security_retirement", value_kind="finite_numeric" + ), + _column( + "person", "social_security_disability", value_kind="finite_numeric" + ), + _column( + "person", "social_security_dependents", value_kind="finite_numeric" + ), + _column( + "person", "social_security_survivors", value_kind="finite_numeric" + ), + ), + ( + _column( + "person", "acs_social_security_income", value_kind="finite_numeric" + ), + ), + optional=True, + ), + _requirement( + "optional_retirement_income", + ( + _column( + "person", + "taxable_private_pension_income", + value_kind="finite_numeric", + ), + _column( + "person", + "tax_exempt_private_pension_income", + value_kind="finite_numeric", + ), + _column( + "person", "taxable_ira_distributions", value_kind="finite_numeric" + ), + ), + (_column("person", "acs_retirement_income", value_kind="finite_numeric"),), + optional=True, + ), + _requirement( + "optional_investment_income", + ( + _column( + "person", "taxable_interest_income", value_kind="finite_numeric" + ), + _column( + "person", "tax_exempt_interest_income", value_kind="finite_numeric" + ), + _column( + "person", "qualified_dividend_income", value_kind="finite_numeric" + ), + _column( + "person", + "non_qualified_dividend_income", + value_kind="finite_numeric", + ), + _column("person", "rental_income", value_kind="finite_numeric"), + _column("person", "estate_income", value_kind="finite_numeric"), + ), + ( + _column( + "person", + "acs_interest_dividend_rental_income", + value_kind="finite_numeric", + ), + ), + optional=True, + ), + _requirement( + "optional_household_head", + (_column("person", "is_household_head", value_kind="finite_numeric"),), + (_column("person", "A_EXPRRP", value_kind="finite_numeric"),), + (_column("person", "A_LINENO", value_kind="finite_numeric"),), + optional=True, + ), + _requirement( + "optional_tenure", + (_column("person", "tenure_type"),), + (_column("spm_unit", "spm_unit_tenure_type"),), + optional=True, + ), + ) + + +def _surface_rows( + surface: TargetFamilies, +) -> tuple[tuple[str, str, tuple[str, ...]], ...]: + entity_order = ( + "person", + "household", + "tax_unit", + "spm_unit", + "family", + "marital_unit", + ) + unknown_entities = sorted(set(surface) - set(entity_order)) + if unknown_entities: + raise RuntimeError( + f"US late transfer surface names unknown entities: {unknown_entities}." + ) + return tuple( + (entity, family, tuple(surface[entity][family])) + for entity in entity_order + if entity in surface + for family in sorted(surface[entity]) + if surface[entity][family] + ) + + +def _bounded_transfer_groups( + surface: TargetFamilies, + *, + max_targets_per_fit: int, +) -> tuple[TransferProducerGroup, ...]: + """Mirror ACS transfer's canonical bounded-family partition.""" + + if max_targets_per_fit <= 0: + raise ValueError("US late transfer max_targets_per_fit must be positive.") + groups: list[TransferProducerGroup] = [] + immigration_pair = ("ssn_card_type", "immigration_status_str") + immigration_set = set(immigration_pair) + for entity, family, targets in _surface_rows(surface): + atoms: list[tuple[str, ...]] = [] + pair_added = False + for target in targets: + if target in immigration_set and immigration_set.issubset(targets): + if not pair_added: + atoms.append(immigration_pair) + pair_added = True + continue + atoms.append((target,)) + batches: list[tuple[str, ...]] = [] + current: list[str] = [] + for atom in atoms: + if current and len(current) + len(atom) > max_targets_per_fit: + batches.append(tuple(current)) + current = [] + current.extend(atom) + if current: + batches.append(tuple(current)) + bounded = ( + ((family, batches[0]),) + if len(batches) == 1 + else tuple( + (f"{family}__batch_{position}", batch) + for position, batch in enumerate(batches, start=1) + ) + ) + for bounded_family, batch in bounded: + target_families: TargetFamilies = MappingProxyType( + {entity: MappingProxyType({bounded_family: batch})} + ) + groups.append( + TransferProducerGroup( + name=transfer_producer_name(entity, bounded_family), + entity=entity, + family=bounded_family, + targets=batch, + target_families=target_families, + ) + ) + return tuple(groups) + + +CANONICAL_US_LATE_TRANSFER_GROUPS = _bounded_transfer_groups( + pool_post_puf_transfer_target_families(), + max_targets_per_fit=_DEFAULT_MAX_TARGETS_PER_FIT, +) +if ( + len(CANONICAL_US_LATE_TRANSFER_GROUPS) != 19 + or sum(len(group.targets) for group in CANONICAL_US_LATE_TRANSFER_GROUPS) != 70 +): + raise RuntimeError( + "Canonical US late transfer must contain exactly 19 bounded groups " + "and 70 ordered targets." + ) +_canonical_late_targets = { + target for group in CANONICAL_US_LATE_TRANSFER_GROUPS for target in group.targets +} +if ( + len(_BOOLEAN_LATE_TARGETS) != 17 + or len(_STRING_LATE_TARGETS) != 2 + or not (_BOOLEAN_LATE_TARGETS | _STRING_LATE_TARGETS) <= _canonical_late_targets + or len(_canonical_late_targets - _BOOLEAN_LATE_TARGETS - _STRING_LATE_TARGETS) != 51 +): + raise RuntimeError( + "Canonical US late target kinds must partition 70 targets into " + "51 numeric, 17 boolean, and 2 string inputs." + ) +US_LATE_TRANSFER_INPUT_INVENTORIES: Mapping[str, SourceInputInventory] = ( + MappingProxyType( + { + group.name: _transfer_input_inventory(group) + for group in CANONICAL_US_LATE_TRANSFER_GROUPS + } + ) +) + + +def _source_outputs() -> dict[str, tuple[ProducerOutput, ...]]: + result: dict[str, tuple[ProducerOutput, ...]] = {} + for operator in POOL_POST_CLONE_SOURCE_OPERATOR_ORDER: + family = POOL_OPERATOR_CONTRACTS[operator].family + outputs = PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES[family] + result[operator] = tuple( + ProducerOutput(entity, column, _ASEC_SOURCE_SCOPE) + for entity in sorted(outputs) + for column in sorted( + set(outputs[entity]) - set(FORMULA_OWNED_SOURCE_COLUMNS.get(entity, ())) + ) + if not ( + operator == "with_us_education_inputs" + and entity == "person" + and column == _QUALIFIED_TUITION + ) + ) + if not result[operator]: + raise RuntimeError( + f"US late source producer {operator!r} owns no persisted output." + ) + return result + + +CANONICAL_US_LATE_SOURCE_OUTPUTS: Mapping[str, tuple[ProducerOutput, ...]] = ( + MappingProxyType(_source_outputs()) +) + + +def _assert_exhaustive_late_overlap_ownership() -> None: + """Fail import unless every permitted multi-producer touch is adjudicated.""" + + primary = { + (entity, column) + for entity, columns in PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES[ + "primary_puf_qrf" + ].items() + for column in columns + } + physical_source_writes = { + (entity, column) + for operator in POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + for entity, columns in PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES[ + POOL_OPERATOR_CONTRACTS[operator].family + ].items() + for column in columns + if column not in FORMULA_OWNED_SOURCE_COLUMNS.get(entity, ()) + } + callback_passthroughs = { + target + for targets in US_LATE_SOURCE_CALLBACK_PASSTHROUGH_OUTPUTS.values() + for target in targets + } + education_passthroughs = { + ("person", column) + for column in set(US_EDUCATION_INPUTS_OUTPUT_COLUMNS) + - set(US_EDUCATION_INPUTS_OWNED_OUTPUT_COLUMNS) + } + if callback_passthroughs != education_passthroughs: + raise RuntimeError( + "Canonical US education callback pass-through inventory changed: " + f"observed={sorted(education_passthroughs)}, " + f"declared={sorted(callback_passthroughs)}." + ) + transfer = { + (group.entity, target) + for group in CANONICAL_US_LATE_TRANSFER_GROUPS + for target in group.targets + } + tail_owned = { + *(("person", column) for column in PUF_CAPITAL_GAINS_TAIL_PERSON_COLUMNS), + *(("tax_unit", column) for column in PUF_CAPITAL_GAINS_TAIL_TAX_UNIT_COLUMNS), + } + recipient_owned = { + *(("person", column) for column in PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS), + *(("tax_unit", column) for column in PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS), + } - tail_owned + declared = set(US_LATE_OVERLAP_OWNERSHIP_TARGETS) + observed = primary & (physical_source_writes | callback_passthroughs) & transfer + observed &= recipient_owned + physical_observed = primary & physical_source_writes & transfer & recipient_owned + expected_physical = declared - {("person", _QUALIFIED_TUITION)} + tail_overlap = ( + primary + & (physical_source_writes | callback_passthroughs) + & transfer + & tail_owned + ) + if observed != declared: + raise RuntimeError( + "Canonical US late overlap ownership does not exhaust the permitted " + f"recipient-owned dual-write surface: observed={sorted(observed)}, " + f"declared={sorted(declared)}." + ) + if physical_observed != expected_physical: + raise RuntimeError( + "Canonical US late physical dual-write surface changed: " + f"observed={sorted(physical_observed)}, " + f"expected={sorted(expected_physical)}." + ) + if tail_overlap: + raise RuntimeError( + "Canonical US late producer DAG permits an unadjudicated tail-owned " + f"dual write: {sorted(tail_overlap)}." + ) + canonical_source_keys = { + (output.entity, output.column) + for outputs in CANONICAL_US_LATE_SOURCE_OUTPUTS.values() + for output in outputs + } + if not expected_physical <= canonical_source_keys or callback_passthroughs & ( + canonical_source_keys + ): + raise RuntimeError( + "Canonical US late source outputs disagree with overlap write/no-op " + "classification." + ) + + +_assert_exhaustive_late_overlap_ownership() + + +def _target_key_rows(surface: TargetFamilies) -> set[tuple[str, str]]: + return { + (entity, target) + for entity, families in surface.items() + for targets in families.values() + for target in targets + } + + +def _inventory_contract_inputs( + node_name: str, + inventory: SourceInputInventory, + *, + required_scope: str, +) -> tuple[ProducerInput, ...]: + """Turn every effective inventory row into an executable gate input.""" + + inputs: list[ProducerInput] = [] + for requirement in inventory.requirements: + first = requirement.alternatives[0][0] + absence_id = f"optional_input:{node_name}:{requirement.label}" + resolved_scope = requirement.required_scope or required_scope + inputs.append( + ProducerInput( + entity=first.entity, + column=f"@effective:{requirement.label}", + required_scope=resolved_scope, + producing_stage=US_LATE_EXTERNAL_STAGES[0], + tolerated_absence_receipts=(absence_id,) + if requirement.optional + else (), + alternatives=tuple( + tuple( + ProducerInputColumn( + item.entity, + item.column, + item.value_kind, + ) + for item in alternative + ) + for alternative in requirement.alternatives + ), + ) + ) + return tuple(inputs) + + +def _build_registry() -> dict[str, ProducerContract]: + late_surface = pool_post_puf_transfer_target_families() + late_keys = _target_key_rows(late_surface) + puf_keys = _target_key_rows(pool_post_puf_puf_producer_target_families()) + declared_primary_outputs = tuple( + ProducerOutput( + entity, + column, + ( + _WHOLE_POOL_SCOPE + if (entity, column) == ("person", "s_corp_income") + else _PUF_CLONE_SCOPE + ), + ) + for entity, columns in PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES[ + "primary_puf_qrf" + ].items() + for column in columns + ) + (ProducerOutput("person", _CLONE_ATTACHMENT_OUTPUT, _WHOLE_POOL_SCOPE),) + structural_outputs = tuple( + ProducerOutput( + column.entity, + column.column, + requirement.required_scope or _WHOLE_POOL_SCOPE, + ) + for requirement in _CROSS_GRAIN_VALIDATION_REQUIREMENTS + for alternative in requirement.alternatives + for column in alternative + if column.entity != "frame" and column.column != "@resolved_weight" + ) + ( + ProducerOutput( + "frame", + _PUF_ATTACHMENT_MANIFEST_INPUT, + _WHOLE_POOL_SCOPE, + ), + *( + ProducerOutput(entity, "@resolved_weight", _WHOLE_POOL_SCOPE) + for entity in _STRUCTURAL_ENTITIES + ), + ) + primary_output_by_key: dict[tuple[str, str], ProducerOutput] = {} + for output in (*declared_primary_outputs, *structural_outputs): + key = (output.entity, output.column) + previous = primary_output_by_key.get(key) + if previous is not None and previous.coverage_scope != output.coverage_scope: + raise RuntimeError( + f"US primary-PUF output {key} has conflicting coverage " + f"{previous.coverage_scope!r} and {output.coverage_scope!r}." + ) + primary_output_by_key[key] = output + primary_outputs = tuple(primary_output_by_key.values()) + primary_keys = {(output.entity, output.column) for output in primary_outputs} + source_owner: dict[tuple[str, str], str] = {} + for operator, outputs in CANONICAL_US_LATE_SOURCE_OUTPUTS.items(): + for output in outputs: + key = (output.entity, output.column) + if key not in late_keys: + continue + previous = source_owner.setdefault(key, operator) + if previous != operator: + raise RuntimeError( + f"US late target {key} has source owners {previous!r} and " + f"{operator!r}." + ) + # Education consumes PUF-owned tuition; it does not produce or pass it + # through. Keep the ownership correction fail-closed here as well as in + # the source output declaration above. + if ("person", _QUALIFIED_TUITION) in source_owner: + raise RuntimeError( + "US education late-source ownership must exclude qualified tuition." + ) + + registry: dict[str, ProducerContract] = {} + registry[US_LATE_ACS_EARNINGS_UNIVERSE_STAGE] = ProducerContract( + name=US_LATE_ACS_EARNINGS_UNIVERSE_STAGE, + kind="acs_earnings_universe", + inputs=_inventory_contract_inputs( + US_LATE_ACS_EARNINGS_UNIVERSE_STAGE, + US_LATE_ACS_EARNINGS_UNIVERSE_INPUT_INVENTORY, + required_scope=_ACS_SOURCE_SCOPE, + ), + outputs=( + *( + ProducerOutput("person", mapped, _ACS_SOURCE_SCOPE) + for mapped in ACS_PUMS_EARNINGS_SOURCE_COLUMNS + ), + ProducerOutput( + "frame", + US_LATE_ACS_EARNINGS_UNIVERSE_RECEIPT_INPUT, + _WHOLE_POOL_SCOPE, + ), + ), + ) + registry[US_LATE_PRIMARY_PUF_STAGE] = ProducerContract( + name=US_LATE_PRIMARY_PUF_STAGE, + kind="primary_puf", + inputs=( + *_inventory_contract_inputs( + US_LATE_PRIMARY_PUF_STAGE, + US_LATE_PRIMARY_PUF_INPUT_INVENTORY, + required_scope=_WHOLE_POOL_SCOPE, + ), + *( + ProducerInput( + "person", + mapped, + _ACS_SOURCE_SCOPE, + US_LATE_ACS_EARNINGS_UNIVERSE_STAGE, + alternatives=( + (ProducerInputColumn("person", mapped, "finite_numeric"),), + ), + ) + for mapped in ACS_PUMS_EARNINGS_SOURCE_COLUMNS + ), + ProducerInput( + "frame", + US_LATE_ACS_EARNINGS_UNIVERSE_RECEIPT_INPUT, + _WHOLE_POOL_SCOPE, + US_LATE_ACS_EARNINGS_UNIVERSE_STAGE, + ), + *( + ProducerInput( + "person", + raw_source, + _ACS_SOURCE_SCOPE, + US_LATE_EXTERNAL_STAGES[0], + alternatives=( + ( + ProducerInputColumn( + "person", + raw_source, + "column_present", + ), + ), + ), + ) + for raw_source in ACS_PUMS_EARNINGS_SOURCE_COLUMNS.values() + ), + ), + outputs=primary_outputs, + ) + + group_by_target = { + (group.entity, target): group + for group in CANONICAL_US_LATE_TRANSFER_GROUPS + for target in group.targets + } + source_dependencies: dict[str, list[ProducerInput]] = defaultdict(list) + source_dependencies["with_us_wic_claim_input"].append( + ProducerInput( + "person", + _PREGNANCY_OUTPUT, + _ASEC_SOURCE_SCOPE, + source_producer_name("with_us_pregnancy_inputs"), + alternatives=( + (ProducerInputColumn("person", _PREGNANCY_OUTPUT, "non_null"),), + ), + ) + ) + source_dependencies["with_us_adult_care_inputs"].extend( + ( + ProducerInput( + "spm_unit", + _CHILDCARE_OUTPUT, + _ASEC_SOURCE_SCOPE, + source_producer_name("with_us_childcare_inputs"), + alternatives=( + ( + ProducerInputColumn( + "spm_unit", + _CHILDCARE_OUTPUT, + "finite_numeric", + ), + ), + ), + ), + ProducerInput( + "person", + _SSTB_EARNED_INCOME, + _ASEC_SOURCE_SCOPE, + group_by_target[("person", _SSTB_EARNED_INCOME)].name, + alternatives=( + ( + ProducerInputColumn( + "person", + _SSTB_EARNED_INCOME, + "finite_numeric", + ), + ), + ), + ), + ) + ) + source_dependencies["with_us_education_inputs"].append( + ProducerInput( + "person", + _QUALIFIED_TUITION, + _ASEC_SOURCE_SCOPE, + group_by_target[("person", _QUALIFIED_TUITION)].name, + alternatives=( + ( + ProducerInputColumn( + "person", + _QUALIFIED_TUITION, + "finite_numeric", + ), + ), + ), + ) + ) + for operator in POOL_POST_CLONE_SOURCE_OPERATOR_ORDER: + name = source_producer_name(operator) + direct_dependencies = [ + *source_dependencies[operator], + ProducerInput( + "person", + _CLONE_ATTACHMENT_OUTPUT, + _WHOLE_POOL_SCOPE, + US_LATE_PRIMARY_PUF_STAGE, + ), + ] + direct_dependency_keys = { + (item.entity, item.column) for item in direct_dependencies + } + for requirement in US_LATE_SOURCE_INPUT_INVENTORIES[operator].requirements: + for alternative in requirement.alternatives: + for item in alternative: + key = (item.entity, item.column) + if key in primary_keys and key not in direct_dependency_keys: + direct_dependencies.append( + ProducerInput( + item.entity, + item.column, + _PUF_CLONE_SCOPE, + US_LATE_PRIMARY_PUF_STAGE, + ) + ) + direct_dependency_keys.add(key) + registry[name] = ProducerContract( + name=name, + kind="post_clone_source", + inputs=tuple( + { + *direct_dependencies, + *_inventory_contract_inputs( + name, + US_LATE_SOURCE_INPUT_INVENTORIES[operator], + required_scope=_ASEC_SOURCE_SCOPE, + ), + } + ), + outputs=( + *CANONICAL_US_LATE_SOURCE_OUTPUTS[operator], + ProducerOutput( + "person", + f"{_SOURCE_RECEIPT_PREFIX}{operator}", + "receipt", + ), + ), + ) + + registry[US_LATE_SOURCE_FINALIZER_STAGE] = ProducerContract( + name=US_LATE_SOURCE_FINALIZER_STAGE, + kind="source_finalizer", + inputs=( + *( + ProducerInput( + "person", + f"{_SOURCE_RECEIPT_PREFIX}{operator}", + _WHOLE_POOL_SCOPE, + source_producer_name(operator), + ) + for operator in POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + ), + ProducerInput( + "person", + US_LATE_SOURCE_FINALIZER_EXECUTION_CONFIG_INPUT, + _WHOLE_POOL_SCOPE, + US_LATE_EXTERNAL_STAGES[0], + ), + ), + outputs=tuple( + ProducerOutput( + declaration["entity"], + column, + _WHOLE_POOL_SCOPE, + ) + for column, declaration in POOL_DEFERRED_TRANSFER_INPUTS.items() + ), + ) + + covered: set[tuple[str, str]] = set() + for group in CANONICAL_US_LATE_TRANSFER_GROUPS: + inputs: list[ProducerInput] = list( + _inventory_contract_inputs( + group.name, + US_LATE_TRANSFER_INPUT_INVENTORIES[group.name], + required_scope=_WHOLE_POOL_SCOPE, + ) + ) + inputs.extend( + ( + ProducerInput( + "person", + "tax_exempt_interest_income", + _PUF_CLONE_SCOPE, + US_LATE_PRIMARY_PUF_STAGE, + alternatives=( + ( + ProducerInputColumn( + "person", + "tax_exempt_interest_income", + "finite_numeric", + ), + ), + ), + ), + ProducerInput( + "person", + "estate_income", + _PUF_CLONE_SCOPE, + US_LATE_PRIMARY_PUF_STAGE, + alternatives=( + ( + ProducerInputColumn( + "person", + "estate_income", + "finite_numeric", + ), + ), + ), + ), + ) + ) + direct_dependency_keys = { + (item.entity, item.column) + for item in inputs + if not item.column.startswith("@effective:") + } + for requirement in US_LATE_TRANSFER_INPUT_INVENTORIES[group.name].requirements: + for alternative in requirement.alternatives: + for item in alternative: + key = (item.entity, item.column) + output = primary_output_by_key.get(key) + if output is None or key in direct_dependency_keys: + continue + inputs.append( + ProducerInput( + item.entity, + item.column, + output.coverage_scope, + US_LATE_PRIMARY_PUF_STAGE, + alternatives=((item,),), + ) + ) + direct_dependency_keys.add(key) + outputs: list[ProducerOutput] = [] + for target in group.targets: + key = (group.entity, target) + covered.add(key) + if key in puf_keys: + inputs.append( + ProducerInput( + group.entity, + target, + _PUF_CLONE_SCOPE, + US_LATE_PRIMARY_PUF_STAGE, + alternatives=( + ( + ProducerInputColumn( + group.entity, + target, + _late_target_value_kind(target), + ), + ), + ), + ) + ) + owner = source_owner.get(key) + if owner is not None: + inputs.append( + ProducerInput( + group.entity, + target, + _ASEC_SOURCE_SCOPE, + source_producer_name(owner), + alternatives=( + ( + ProducerInputColumn( + group.entity, + target, + _late_target_value_kind(target), + ), + ), + ), + ) + ) + if key not in puf_keys and owner is None: + raise RuntimeError( + f"US late transfer target {key} has no declared producer." + ) + outputs.append(ProducerOutput(group.entity, target, _WHOLE_POOL_SCOPE)) + registry[group.name] = ProducerContract( + name=group.name, + kind="late_transfer", + inputs=tuple(set(inputs)), + outputs=tuple(outputs), + ) + if covered != late_keys: + raise RuntimeError( + "US late transfer groups do not exactly cover the canonical surface; " + f"missing={sorted(late_keys - covered)}, extra={sorted(covered - late_keys)}." + ) + return registry + + +CANONICAL_US_LATE_PRODUCER_REGISTRY: Mapping[str, ProducerContract] = MappingProxyType( + _build_registry() +) +CANONICAL_US_LATE_PRODUCER_SCHEDULE: ProducerSchedule = derive_producer_schedule( + CANONICAL_US_LATE_PRODUCER_REGISTRY, + external_stages=US_LATE_EXTERNAL_STAGES, +) + + +def _inventory_payload(inventory: SourceInputInventory) -> dict[str, object]: + return { + "operator": inventory.operator, + "requirements": [ + { + "label": requirement.label, + "optional": requirement.optional, + "required_scope": requirement.required_scope, + "alternatives": [ + [ + { + "entity": item.entity, + "column": item.column, + "value_kind": item.value_kind, + } + for item in alternative + ] + for alternative in requirement.alternatives + ], + } + for requirement in inventory.requirements + ], + } + + +def us_late_producer_schedule_payload() -> dict[str, object]: + """Return the complete JSON-safe declaration bound into checkpoint identity.""" + + schedule = CANONICAL_US_LATE_PRODUCER_SCHEDULE + return { + "schema_version": US_LATE_PRODUCER_REGISTRY_SCHEMA_VERSION, + "overlap_ownership": dict(us_late_overlap_ownership_receipt()), + "execution_receipt_contract": { + "version": US_LATE_PRODUCER_RECEIPT_SCHEMA_VERSION, + "row_binding": ( + "declared_globally_reconciled_input_and_scope_exact_output_" + "source_and_primary_callback_resource_receipt_and_previous_" + "execution_sha256" + ), + "virtual_resource_binding": ( + "exact_kind_specific_semantic_payload_and_sha256" + ), + "top_binding": ( + "entry_and_output_frame_sha256_execution_chain_source_" + "completion_and_nineteen_transfer_groups" + ), + "transition_authority": { + "authority_id": US_LATE_PRODUCER_TRANSITION_AUTHORITY_ID, + "metadata_key": US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY, + "version": US_LATE_PRODUCER_TRANSITION_AUTHORITY_VERSION, + "independent_digest_required": True, + }, + }, + "schedule_sha256": schedule.sha256, + "external_stages": list(US_LATE_EXTERNAL_STAGES), + "order": list(schedule.order), + "waves": [list(wave) for wave in schedule.waves], + "edges": [list(edge) for edge in schedule.edges], + "transfer_groups": [ + { + "name": group.name, + "entity": group.entity, + "family": group.family, + "targets": list(group.targets), + } + for group in CANONICAL_US_LATE_TRANSFER_GROUPS + ], + "source_input_inventories": [ + _inventory_payload(US_LATE_SOURCE_INPUT_INVENTORIES[operator]) + for operator in sorted(US_LATE_SOURCE_INPUT_INVENTORIES) + ], + "primary_puf_input_inventory": _inventory_payload( + US_LATE_PRIMARY_PUF_INPUT_INVENTORY + ), + "acs_earnings_universe_input_inventory": _inventory_payload( + US_LATE_ACS_EARNINGS_UNIVERSE_INPUT_INVENTORY + ), + "transfer_input_inventories": [ + _inventory_payload(US_LATE_TRANSFER_INPUT_INVENTORIES[name]) + for name in sorted(US_LATE_TRANSFER_INPUT_INVENTORIES) + ], + } + + +def us_late_producer_schedule_receipt() -> Mapping[str, object]: + """Return the byte-stable production schedule identity and useful counts.""" + + payload = us_late_producer_schedule_payload() + canonical = json.dumps( + payload, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + return MappingProxyType( + { + **payload, + "payload_sha256": hashlib.sha256(canonical).hexdigest(), + "producer_count": len(CANONICAL_US_LATE_PRODUCER_REGISTRY), + "source_producer_count": len(POOL_POST_CLONE_SOURCE_OPERATOR_ORDER), + "transfer_group_count": len(CANONICAL_US_LATE_TRANSFER_GROUPS), + "transfer_target_count": sum( + len(group.targets) for group in CANONICAL_US_LATE_TRANSFER_GROUPS + ), + "status": "derived_and_import_validated", + } + ) diff --git a/packages/microcosm-build/tests/test_frame_checkpoint.py b/packages/microcosm-build/tests/test_frame_checkpoint.py index 8e8c5553..7c79ba3f 100644 --- a/packages/microcosm-build/tests/test_frame_checkpoint.py +++ b/packages/microcosm-build/tests/test_frame_checkpoint.py @@ -1,5 +1,6 @@ from __future__ import annotations +import json import os import stat import time @@ -141,6 +142,66 @@ def _checkpoint_frame() -> Frame: ) +def _nullable_boolean_checkpoint_frame() -> Frame: + frame = _checkpoint_frame() + tables = {name: frame.table(name).copy() for name in frame.entities} + tables.update({name: frame.link(name).copy() for name in frame.links}) + person = tables["person"] + person["complete_nullable_boolean"] = pd.Series( + [True, False, True], + index=person.index, + dtype="boolean", + ) + person["missing_nullable_boolean"] = pd.Series( + [True, pd.NA, False], + index=person.index, + dtype="boolean", + ) + jobs = tables["jobs"] + jobs["link_nullable_boolean"] = pd.Series( + [pd.NA, False, True, pd.NA], + index=jobs.index, + dtype="boolean", + ) + return Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + ) + + +def _checkpoint_series_group(root, *, table: str, column: str): + metadata = json.loads(np.asarray(root["metadata_json"]).tobytes()) + table_position = next( + position + for position, spec in enumerate(metadata["tables"]) + if spec["name"] == table + ) + table_spec = metadata["tables"][table_position] + column_position = next( + position + for position, spec in enumerate(table_spec["columns"]) + if spec["name"] == column + ) + return ( + metadata, + table_spec["columns"][column_position], + root[f"tables/t{table_position:05d}/columns/c{column_position:05d}"], + ) + + +def _replace_checkpoint_metadata(root, metadata: dict[str, object]) -> None: + encoded = frame_checkpoint_module._canonical_json(metadata).encode("utf-8") + del root["metadata_json"] + root.create_dataset( + "metadata_json", + data=np.frombuffer(encoded, dtype=np.uint8), + track_times=False, + ) + + def test_frame_checkpoint_round_trip_is_byte_identical(tmp_path: Path) -> None: frame = _checkpoint_frame() first_path = tmp_path / "first.h5" @@ -209,6 +270,312 @@ def test_frame_checkpoint_round_trip_is_byte_identical(tmp_path: Path) -> None: ) +def test_frame_without_nullable_booleans_keeps_schema_2_byte_golden( + tmp_path: Path, +) -> None: + path = tmp_path / "legacy-schema-2.h5" + + write_frame_checkpoint(path, _checkpoint_frame()) + + # No cross-platform byte constant: HDF5 wheels differ between macOS and + # Linux, so whole-file hashes are platform-dependent (run-stable only). + # The contract is determinism plus staying on schema 2 for frames with + # no nullable booleans — assert exactly that. + rewrite = tmp_path / "legacy-schema-2-rewrite.h5" + write_frame_checkpoint(rewrite, _checkpoint_frame()) + assert path.read_bytes() == rewrite.read_bytes() + h5py = pytest.importorskip("h5py") + with h5py.File(path, mode="r") as h5: + raw = np.asarray(h5["_populace_frame_checkpoint/metadata_json"]).tobytes() + assert json.loads(raw.decode("utf-8"))["schema_version"] == 2 + + +def test_nullable_boolean_round_trip_preserves_dtype_values_and_null_masks( + tmp_path: Path, +) -> None: + frame = _nullable_boolean_checkpoint_frame() + first_path = tmp_path / "nullable-first.h5" + second_path = tmp_path / "nullable-second.h5" + + write_frame_checkpoint(first_path, frame) + loaded = load_frame_checkpoint(first_path) + write_frame_checkpoint(second_path, loaded.frame) + + assert first_path.read_bytes() == second_path.read_bytes() + for table, column in ( + ("person", "complete_nullable_boolean"), + ("person", "missing_nullable_boolean"), + ("jobs", "link_nullable_boolean"), + ): + loaded_table = ( + loaded.frame.table(table) + if table in loaded.frame.entities + else loaded.frame.link(table) + ) + expected_table = ( + frame.table(table) if table in frame.entities else frame.link(table) + ) + pd.testing.assert_series_equal( + loaded_table[column], + expected_table[column], + check_dtype=True, + check_exact=True, + ) + assert loaded_table[column].dtype == pd.BooleanDtype() + + h5py = pytest.importorskip("h5py") + with h5py.File(first_path, mode="r") as h5: + root = h5["_populace_frame_checkpoint"] + metadata = json.loads(np.asarray(root["metadata_json"]).tobytes()) + assert metadata["schema_version"] == 3 + for table, column, expected_mask in ( + ("person", "complete_nullable_boolean", None), + ("person", "missing_nullable_boolean", [0, 1, 0]), + ("jobs", "link_nullable_boolean", [1, 0, 0, 1]), + ): + _metadata, spec, group = _checkpoint_series_group( + root, + table=table, + column=column, + ) + assert spec == { + "name": column, + "dtype": "boolean", + "encoding": "nullable_boolean_v1", + "has_null_mask": expected_mask is not None, + } + assert np.asarray(group["values"]).dtype == np.dtype(np.bool_) + if expected_mask is None: + assert "null_mask" not in group + else: + mask = np.asarray(group["null_mask"]) + assert mask.dtype == np.dtype(np.uint8) + assert mask.tolist() == expected_mask + + +def test_nullable_boolean_masked_storage_bits_are_canonical( + tmp_path: Path, +) -> None: + false_hidden = _nullable_boolean_checkpoint_frame() + true_hidden = _nullable_boolean_checkpoint_frame() + false_person = false_hidden.person + true_person = true_hidden.person + false_person["missing_nullable_boolean"] = pd.Series( + pd.arrays.BooleanArray( + np.asarray([True, False, False], dtype=np.bool_), + np.asarray([False, True, False], dtype=np.bool_), + ), + index=false_person.index, + ) + true_person["missing_nullable_boolean"] = pd.Series( + pd.arrays.BooleanArray( + np.asarray([True, True, False], dtype=np.bool_), + np.asarray([False, True, False], dtype=np.bool_), + ), + index=true_person.index, + ) + first_path = tmp_path / "hidden-false.h5" + second_path = tmp_path / "hidden-true.h5" + + write_frame_checkpoint(first_path, false_hidden) + write_frame_checkpoint(second_path, true_hidden) + + assert first_path.read_bytes() == second_path.read_bytes() + + +@pytest.mark.parametrize( + "damage", + [ + "missing", + "all_zero", + "nonbinary", + "wrong_length", + "wrong_dtype", + "wrong_rank", + ], +) +def test_nullable_boolean_checkpoint_rejects_malformed_null_mask( + tmp_path: Path, + damage: str, +) -> None: + path = tmp_path / f"malformed-{damage}.h5" + write_frame_checkpoint(path, _nullable_boolean_checkpoint_frame()) + h5py = pytest.importorskip("h5py") + with h5py.File(path, mode="r+") as h5: + root = h5["_populace_frame_checkpoint"] + _metadata, _spec, group = _checkpoint_series_group( + root, + table="person", + column="missing_nullable_boolean", + ) + del group["null_mask"] + if damage == "all_zero": + group.create_dataset( + "null_mask", + data=np.asarray([0, 0, 0], dtype=np.uint8), + track_times=False, + ) + elif damage == "nonbinary": + group.create_dataset( + "null_mask", + data=np.asarray([0, 2, 0], dtype=np.uint8), + track_times=False, + ) + elif damage == "wrong_length": + group.create_dataset( + "null_mask", + data=np.asarray([0, 1], dtype=np.uint8), + track_times=False, + ) + elif damage == "wrong_dtype": + group.create_dataset( + "null_mask", + data=np.asarray([0, 1, 0], dtype=np.int16), + track_times=False, + ) + elif damage == "wrong_rank": + group.create_dataset( + "null_mask", + data=np.asarray([[0, 1, 0]], dtype=np.uint8), + track_times=False, + ) + + with pytest.raises(ValueError, match="null mask"): + load_frame_checkpoint(path) + + +@pytest.mark.parametrize("damage", ["wrong_dtype", "wrong_rank"]) +def test_nullable_boolean_checkpoint_rejects_malformed_values( + tmp_path: Path, + damage: str, +) -> None: + path = tmp_path / f"malformed-values-{damage}.h5" + write_frame_checkpoint(path, _nullable_boolean_checkpoint_frame()) + h5py = pytest.importorskip("h5py") + with h5py.File(path, mode="r+") as h5: + root = h5["_populace_frame_checkpoint"] + _metadata, _spec, group = _checkpoint_series_group( + root, + table="person", + column="missing_nullable_boolean", + ) + del group["values"] + values = ( + np.asarray([1, 0, 0], dtype=np.uint8) + if damage == "wrong_dtype" + else np.asarray([[True, False, False]], dtype=np.bool_) + ) + group.create_dataset("values", data=values, track_times=False) + + with pytest.raises(ValueError, match="nullable boolean values"): + load_frame_checkpoint(path) + + +def test_maskless_nullable_boolean_rejects_unexpected_null_mask( + tmp_path: Path, +) -> None: + path = tmp_path / "unexpected-mask.h5" + write_frame_checkpoint(path, _nullable_boolean_checkpoint_frame()) + h5py = pytest.importorskip("h5py") + with h5py.File(path, mode="r+") as h5: + root = h5["_populace_frame_checkpoint"] + _metadata, _spec, group = _checkpoint_series_group( + root, + table="person", + column="complete_nullable_boolean", + ) + group.create_dataset( + "null_mask", + data=np.zeros(3, dtype=np.uint8), + track_times=False, + ) + + with pytest.raises(ValueError, match="unexpected null mask"): + load_frame_checkpoint(path) + + +@pytest.mark.parametrize( + ("field", "value", "message"), + [ + ("has_null_mask", "yes", "has_null_mask"), + ("dtype", "bool", "declared dtype"), + ], +) +def test_nullable_boolean_checkpoint_rejects_malformed_spec( + tmp_path: Path, + field: str, + value: object, + message: str, +) -> None: + path = tmp_path / f"malformed-spec-{field}.h5" + write_frame_checkpoint(path, _nullable_boolean_checkpoint_frame()) + h5py = pytest.importorskip("h5py") + with h5py.File(path, mode="r+") as h5: + root = h5["_populace_frame_checkpoint"] + metadata, spec, _group = _checkpoint_series_group( + root, + table="person", + column="missing_nullable_boolean", + ) + spec[field] = value + _replace_checkpoint_metadata(root, metadata) + + with pytest.raises(ValueError, match=message): + load_frame_checkpoint(path) + + +def test_schema_2_checkpoint_rejects_boolean_dtype_under_legacy_encoding( + tmp_path: Path, +) -> None: + path = tmp_path / "forged-schema-2-object-boolean.h5" + write_frame_checkpoint(path, _checkpoint_frame()) + h5py = pytest.importorskip("h5py") + with h5py.File(path, mode="r+") as h5: + root = h5["_populace_frame_checkpoint"] + metadata, spec, _group = _checkpoint_series_group( + root, + table="person", + column="nullable_flag", + ) + spec["dtype"] = "boolean" + _replace_checkpoint_metadata(root, metadata) + + with pytest.raises(ValueError, match="schema version 2.*boolean"): + load_frame_checkpoint(path) + + +def test_schema_2_checkpoint_cannot_smuggle_nullable_boolean_encoding( + tmp_path: Path, +) -> None: + path = tmp_path / "forged-schema-2.h5" + write_frame_checkpoint(path, _nullable_boolean_checkpoint_frame()) + h5py = pytest.importorskip("h5py") + with h5py.File(path, mode="r+") as h5: + root = h5["_populace_frame_checkpoint"] + metadata = json.loads(np.asarray(root["metadata_json"]).tobytes()) + metadata["schema_version"] = 2 + _replace_checkpoint_metadata(root, metadata) + + with pytest.raises(ValueError, match="schema version 2.*nullable"): + load_frame_checkpoint(path) + + +def test_schema_3_checkpoint_requires_nullable_boolean_encoding( + tmp_path: Path, +) -> None: + path = tmp_path / "forged-schema-3.h5" + write_frame_checkpoint(path, _checkpoint_frame()) + h5py = pytest.importorskip("h5py") + with h5py.File(path, mode="r+") as h5: + root = h5["_populace_frame_checkpoint"] + metadata = json.loads(np.asarray(root["metadata_json"]).tobytes()) + metadata["schema_version"] = 3 + _replace_checkpoint_metadata(root, metadata) + + with pytest.raises(ValueError, match="schema version 3.*nullable"): + load_frame_checkpoint(path) + + def test_frame_checkpoint_fsyncs_parent_directory_after_rename( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, @@ -290,10 +657,9 @@ def test_frame_checkpoint_preserves_range_indexes_and_column_axis_name( "unsupported", [ pd.Series([1, pd.NA, 3], dtype="Int64"), - pd.Series([True, pd.NA, False], dtype="boolean"), pd.Series(["a", "b", "a"], dtype="category"), ], - ids=["nullable_integer", "nullable_boolean", "categorical"], + ids=["nullable_integer", "categorical"], ) def test_frame_checkpoint_rejects_unsupported_dtype_without_replacing_destination( tmp_path: Path, diff --git a/packages/microcosm-build/tests/test_frame_serializer_registry.py b/packages/microcosm-build/tests/test_frame_serializer_registry.py new file mode 100644 index 00000000..04f5df7d --- /dev/null +++ b/packages/microcosm-build/tests/test_frame_serializer_registry.py @@ -0,0 +1,532 @@ +from __future__ import annotations + +import ast +import importlib.util +import json +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path + +import numpy as np +import pandas as pd +import pytest + +from microcosm.build.frame_checkpoint import ( + load_frame_checkpoint, + write_frame_checkpoint, +) +from microcosm.build.frame_serializer_registry import ( + FRAME_TABLE_SERIALIZERS, + HDF_WRITE_EXCLUSIONS, + FrameSerializerSpec, +) +from microcosm.build.uk_runtime.national_build import ( + _read_uk_national_tables, + _write_uk_single_year_tables, +) +from microcosm.build.us_runtime.h5_io import write_nullable_us_h5 +from microcosm.frame import ( + US_SCHEMA, + EntitySchema, + Frame, + WeightKind, + Weights, + read_frame_table, +) +from microcosm.frame.adapters.axiom import AxiomEntityTableDataset +from microcosm.frame.adapters.policyengine_us import PolicyEngineUSEngine + +REPOSITORY_ROOT = Path(__file__).resolve().parents[3] +PRODUCTION_ROOTS = ( + REPOSITORY_ROOT / "packages", + REPOSITORY_ROOT / "tools", +) +COMPLETE_COLUMN = "fixture_complete_nullable_boolean" +MISSING_COLUMN = "fixture_missing_nullable_boolean" +NATIVE_COLUMN = "fixture_native_boolean" + + +@dataclass(frozen=True) +class BooleanRoundTrip: + source: pd.DataFrame + source_before_write: pd.DataFrame + loaded: pd.DataFrame + stored_complete_values: np.ndarray + stored_missing_values: np.ndarray + stored_missing_mask: np.ndarray + stored_missing_mask_dtype: np.dtype | None + + +RoundTripAdapter = Callable[[Path, str], BooleanRoundTrip] + + +def _dtype_family_table( + nullable_case: str, + *, + id_column: str = "person_id", +) -> pd.DataFrame: + index = pd.RangeIndex(3) + missing_values = { + "mixed": [True, pd.NA, False], + "all_missing": [pd.NA, pd.NA, pd.NA], + }[nullable_case] + return pd.DataFrame( + { + id_column: np.asarray([1, 2, 3], dtype=np.int64), + NATIVE_COLUMN: np.asarray([False, True, False], dtype=np.bool_), + COMPLETE_COLUMN: pd.Series( + [True, False, True], index=index, dtype="boolean" + ), + MISSING_COLUMN: pd.Series(missing_values, index=index, dtype="boolean"), + }, + index=index, + ) + + +def _semantic_observation( + source: pd.DataFrame, + source_before_write: pd.DataFrame, + loaded: pd.DataFrame, +) -> BooleanRoundTrip: + missing = loaded[MISSING_COLUMN] + return BooleanRoundTrip( + source=source, + source_before_write=source_before_write, + loaded=loaded, + stored_complete_values=loaded[COMPLETE_COLUMN].to_numpy( + dtype=np.bool_, copy=False + ), + stored_missing_values=missing.to_numpy( + dtype=np.bool_, na_value=False, copy=False + ), + stored_missing_mask=missing.isna().to_numpy(dtype=np.bool_, copy=False), + stored_missing_mask_dtype=None, + ) + + +def _small_frame(source: pd.DataFrame) -> Frame: + household = pd.DataFrame( + {"household_id": np.asarray([1, 2, 3], dtype=np.int64)}, + index=source.index, + ) + person = source.copy(deep=False) + person.insert( + 1, + "person_household_id", + np.asarray([1, 2, 3], dtype=np.int64), + ) + return Frame( + {"person": person, "household": household}, + EntitySchema(group_entities=("household",)), + { + "household": Weights( + np.asarray([1.0, 2.0, 3.0]), + WeightKind.DESIGN, + ) + }, + ) + + +def _us_frame(source: pd.DataFrame) -> Frame: + ids = np.asarray([1, 2, 3], dtype=np.int64) + person = source.copy(deep=False) + for position, entity in enumerate(US_SCHEMA.group_entities, start=1): + person.insert(position, US_SCHEMA.membership_column(entity), ids) + tables = { + "person": person, + **{ + entity: pd.DataFrame({US_SCHEMA.id_column(entity): ids}, index=source.index) + for entity in US_SCHEMA.group_entities + }, + } + return Frame( + tables, + US_SCHEMA, + {"household": Weights(np.asarray([1.0, 2.0, 3.0]), WeightKind.DESIGN)}, + ) + + +def _load_tool(relative_path: str, module_name: str): + path = REPOSITORY_ROOT / relative_path + spec = importlib.util.spec_from_file_location(module_name, path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _checkpoint_column_group(root, *, table: str, column: str): + metadata = json.loads(np.asarray(root["metadata_json"]).tobytes()) + table_index, table_spec = next( + (index, spec) + for index, spec in enumerate(metadata["tables"]) + if spec["name"] == table + ) + column_index = next( + index + for index, spec in enumerate(table_spec["columns"]) + if spec["name"] == column + ) + return root["tables"][f"t{table_index:05d}"]["columns"][f"c{column_index:05d}"] + + +def _round_trip_frame_checkpoint( + tmp_path: Path, nullable_case: str +) -> BooleanRoundTrip: + h5py = pytest.importorskip("h5py") + source = _dtype_family_table(nullable_case) + before = source.copy(deep=True) + frame = _small_frame(source) + path = tmp_path / "frame-checkpoint.h5" + write_frame_checkpoint(path, frame) + loaded = load_frame_checkpoint(path).frame.table("person") + with h5py.File(path, mode="r") as h5: + root = h5["_populace_frame_checkpoint"] + complete = _checkpoint_column_group( + root, table="person", column=COMPLETE_COLUMN + ) + missing = _checkpoint_column_group(root, table="person", column=MISSING_COLUMN) + return BooleanRoundTrip( + source=source, + source_before_write=before, + loaded=loaded, + stored_complete_values=np.asarray(complete["values"]), + stored_missing_values=np.asarray(missing["values"]), + stored_missing_mask=np.asarray(missing["null_mask"], dtype=np.bool_), + stored_missing_mask_dtype=np.asarray(missing["null_mask"]).dtype, + ) + + +def _round_trip_nullable_us_h5(tmp_path: Path, nullable_case: str) -> BooleanRoundTrip: + pytest.importorskip("tables") + source = _dtype_family_table(nullable_case) + before = source.copy(deep=True) + frame = _us_frame(source) + path = tmp_path / "nullable-us.h5" + write_nullable_us_h5( + frame, + path, + period=2024, + artifact_kind="registry_dtype_family_fixture", + ) + with pd.HDFStore(path, mode="r") as store: + loaded = read_frame_table(store, "person") + return _semantic_observation(source, before, loaded) + + +def _round_trip_uk_single_year(tmp_path: Path, nullable_case: str) -> BooleanRoundTrip: + pytest.importorskip("tables") + pytest.importorskip("h5py") + source = _dtype_family_table(nullable_case) + before = source.copy(deep=True) + path = tmp_path / "uk-single-year.h5" + _write_uk_single_year_tables( + person=source, + benunit=pd.DataFrame({"benunit_id": [1, 2, 3]}, index=source.index), + household=pd.DataFrame( + { + "household_id": [1, 2, 3], + "household_weight": [1.0, 2.0, 3.0], + }, + index=source.index, + ), + time_period="2023", + weight_kind=WeightKind.DESIGN, + mass_log=(), + path=path, + ) + loaded = _read_uk_national_tables(path)[0]["person"] + return _semantic_observation(source, before, loaded) + + +def _round_trip_axiom(tmp_path: Path, nullable_case: str) -> BooleanRoundTrip: + pytest.importorskip("tables") + source = _dtype_family_table(nullable_case) + before = source.copy(deep=True) + path = tmp_path / "axiom.h5" + AxiomEntityTableDataset(tables={"person": source}, time_period=2025).save(path) + loaded = AxiomEntityTableDataset(file_path=path).person + return _semantic_observation(source, before, loaded) + + +def _round_trip_policyengine_us(tmp_path: Path, nullable_case: str) -> BooleanRoundTrip: + pytest.importorskip("tables") + pytest.importorskip("policyengine_us") + source = _dtype_family_table(nullable_case) + before = source.copy(deep=True) + frame = _us_frame(source) + tables = {entity: frame.table(entity) for entity in frame.entities} + path = tmp_path / "policyengine-us.h5" + PolicyEngineUSEngine()._write_and_verify(tables, period=2024, output_path=path) + with pd.HDFStore(path, mode="r") as store: + loaded = read_frame_table(store, "person") + return _semantic_observation(source, before, loaded) + + +def _round_trip_legacy_us(tmp_path: Path, nullable_case: str) -> BooleanRoundTrip: + pytest.importorskip("tables") + legacy = _load_tool( + "tools/_legacy/build_us_acs_multispine_base.py", + "registry_legacy_us_builder", + ) + source = _dtype_family_table(nullable_case) + before = source.copy(deep=True) + path = tmp_path / "legacy-us.h5" + legacy._write_dataset(_us_frame(source), path, period=2024) + with pd.HDFStore(path, mode="r") as store: + loaded = read_frame_table(store, "person") + return _semantic_observation(source, before, loaded) + + +def _round_trip_acs_lean(tmp_path: Path, nullable_case: str) -> BooleanRoundTrip: + pytest.importorskip("tables") + tool = _load_tool( + "tools/build_us_acs_local_release.py", + "registry_acs_local_release", + ) + source = _dtype_family_table(nullable_case) + before = source.copy(deep=True) + ids = np.asarray([1, 2, 3], dtype=np.int64) + person = source.copy(deep=False) + for position, entity in enumerate(US_SCHEMA.group_entities, start=1): + person.insert(position, US_SCHEMA.membership_column(entity), ids) + struct = { + "household_struct": pd.DataFrame({"household_id": ids}), + "person": person, + "groups": { + entity: pd.DataFrame({US_SCHEMA.id_column(entity): ids}) + for entity in tool.GROUP_IDS + }, + "weights": np.asarray([1.0, 2.0, 3.0]), + } + path, _targets = tool.write_lean_checkpoint( + struct, + np.empty((3, 0), dtype=np.float64), + [], + [], + [], + [], + [], + tmp_path / "acs-lean", + ) + loaded = tool.load_lean_frame(path)[0].table("person") + return _semantic_observation(source, before, loaded) + + +def _round_trip_fiscal_checkpoint( + tmp_path: Path, nullable_case: str +) -> BooleanRoundTrip: + h5py = pytest.importorskip("h5py") + tool = _load_tool( + "tools/build_us_fiscal_refresh_release.py", + "registry_fiscal_refresh", + ) + source = _dtype_family_table(nullable_case) + before = source.copy(deep=True) + frame = _small_frame(source) + path = tmp_path / "fiscal-target-frame.h5" + tool._write_target_frame_checkpoint( + path, + frame=frame, + identity={"registry_fixture": True}, + compilation={}, + ) + with h5py.File(path, mode="r") as h5: + person = h5["tables"]["person"] + loaded = tool._read_checkpoint_dataframe(person) + columns = json.loads(str(person.attrs["columns_json"])) + complete_index = columns.index(COMPLETE_COLUMN) + missing_index = columns.index(MISSING_COLUMN) + complete = person["columns"][f"{complete_index:05d}"] + missing = person["columns"][f"{missing_index:05d}"] + return BooleanRoundTrip( + source=source, + source_before_write=before, + loaded=loaded, + stored_complete_values=np.asarray(complete["values"]), + stored_missing_values=np.asarray(missing["values"]), + stored_missing_mask=np.asarray(missing["null_mask"], dtype=np.bool_), + stored_missing_mask_dtype=np.asarray(missing["null_mask"]).dtype, + ) + + +ROUND_TRIP_ADAPTERS: dict[str, RoundTripAdapter] = { + "frame_checkpoint": _round_trip_frame_checkpoint, + "nullable_us_h5": _round_trip_nullable_us_h5, + "uk_single_year_h5": _round_trip_uk_single_year, + "axiom_entity_tables": _round_trip_axiom, + "policyengine_us_single_year": _round_trip_policyengine_us, + "legacy_us_two_spine": _round_trip_legacy_us, + "acs_local_lean_checkpoint": _round_trip_acs_lean, + "fiscal_target_frame_checkpoint": _round_trip_fiscal_checkpoint, +} + + +def _qualified_call_name(call: ast.Call) -> str | None: + if not isinstance(call.func, ast.Attribute): + return None + if not isinstance(call.func.value, ast.Name): + return None + return f"{call.func.value.id}.{call.func.attr}" + + +def _enclosing_function( + node: ast.AST, + parents: dict[ast.AST, ast.AST], +) -> str: + current = node + while current in parents: + current = parents[current] + if isinstance(current, (ast.FunctionDef, ast.AsyncFunctionDef)): + return current.name + return "" + + +def _literal_hdf_mode(call: ast.Call) -> str: + mode_node: ast.AST | None = call.args[1] if len(call.args) > 1 else None + for keyword in call.keywords: + if keyword.arg == "mode": + mode_node = keyword.value + break + if mode_node is None: + return "a" + value = ast.literal_eval(mode_node) + if not isinstance(value, str): + raise AssertionError("Production HDF modes must be literal strings.") + return value + + +def _discover_writable_hdf_sites() -> set[str]: + discovered: set[str] = set() + for root in PRODUCTION_ROOTS: + for path in root.rglob("*.py"): + if "tests" in path.parts: + continue + tree = ast.parse(path.read_text()) + parents: dict[ast.AST, ast.AST] = {} + for parent in ast.walk(tree): + for child in ast.iter_child_nodes(parent): + parents[child] = parent + for node in ast.walk(tree): + if not isinstance(node, ast.Call): + continue + if _qualified_call_name(node) not in {"h5py.File", "pd.HDFStore"}: + continue + if _literal_hdf_mode(node) == "r": + continue + relative = path.relative_to(REPOSITORY_ROOT).as_posix() + function = _enclosing_function(node, parents) + discovered.add(f"{relative}::{function}") + return discovered + + +def test_registry_classifies_every_writable_production_hdf_site() -> None: + classified = { + spec.writer.key for spec in FRAME_TABLE_SERIALIZERS if spec.direct_hdf_open + } + classified.update(exclusion.writer.key for exclusion in HDF_WRITE_EXCLUSIONS) + assert _discover_writable_hdf_sites() == classified + + +def test_registry_has_exactly_eight_unique_frame_table_serializers() -> None: + assert len(FRAME_TABLE_SERIALIZERS) == 8 + assert len({spec.serializer_id for spec in FRAME_TABLE_SERIALIZERS}) == 8 + assert len({spec.writer.key for spec in FRAME_TABLE_SERIALIZERS}) == 8 + + +def test_round_trip_adapter_registry_exactly_matches_serializer_registry() -> None: + assert set(ROUND_TRIP_ADAPTERS) == { + spec.serializer_id for spec in FRAME_TABLE_SERIALIZERS + } + + +@pytest.mark.parametrize( + "serializer", + FRAME_TABLE_SERIALIZERS, + ids=lambda serializer: serializer.serializer_id, +) +@pytest.mark.parametrize("nullable_case", ("mixed", "all_missing")) +def test_registered_serializer_round_trips_nullable_boolean_dtype_family( + serializer: FrameSerializerSpec, + nullable_case: str, + tmp_path: Path, +) -> None: + observation = ROUND_TRIP_ADAPTERS[serializer.serializer_id](tmp_path, nullable_case) + + # Serializers may materialize a boundary copy, never rewrite the source. + pd.testing.assert_frame_equal( + observation.source, + observation.source_before_write, + check_exact=True, + check_dtype=True, + ) + + loaded = observation.loaded + assert loaded[NATIVE_COLUMN].dtype == np.dtype(np.bool_) + assert loaded[NATIVE_COLUMN].to_numpy(copy=False).tobytes() == ( + observation.source[NATIVE_COLUMN].to_numpy(copy=False).tobytes() + ) + + # Complete BooleanDtype columns have exactly the same physical bool bytes + # as their logical values; PyTables-facing codecs reload them as native + # bool, while the explicit h5py codecs retain semantic dtype metadata. + complete_expected = observation.source[COMPLETE_COLUMN].to_numpy( + dtype=np.bool_, copy=False + ) + assert observation.stored_complete_values.dtype == np.dtype(np.bool_) + assert observation.stored_complete_values.tobytes() == complete_expected.tobytes() + + # Missing booleans are canonical false value bits plus an explicit null + # representation. The semantic reload must recover both observations and + # the exact NA positions. + missing_expected = observation.source[MISSING_COLUMN] + expected_values = missing_expected.to_numpy( + dtype=np.bool_, na_value=False, copy=False + ) + expected_mask = missing_expected.isna().to_numpy(dtype=np.bool_, copy=False) + assert observation.stored_missing_values.dtype == np.dtype(np.bool_) + assert observation.stored_missing_values.tobytes() == expected_values.tobytes() + np.testing.assert_array_equal(observation.stored_missing_mask, expected_mask) + pd.testing.assert_series_equal( + loaded[MISSING_COLUMN].astype("boolean"), + missing_expected, + check_names=True, + ) + + if serializer.nullable_boolean_storage == "bool_values_optional_uint8_mask": + assert observation.stored_missing_mask_dtype == np.dtype(np.uint8) + else: + assert serializer.nullable_boolean_storage == "numpy_bool_or_object_pd_na_v1" + assert loaded[COMPLETE_COLUMN].dtype == np.dtype(np.bool_) + assert loaded[MISSING_COLUMN].dtype == np.dtype(object) + missing_scalars = loaded.loc[loaded[MISSING_COLUMN].isna(), MISSING_COLUMN] + assert all(value is pd.NA for value in missing_scalars) + + +def test_policyengine_us_adapter_owns_its_registered_hdf_boundary() -> None: + (spec,) = ( + candidate + for candidate in FRAME_TABLE_SERIALIZERS + if candidate.serializer_id == "policyengine_us_single_year" + ) + assert spec.direct_hdf_open is True + assert spec.writer.key in _discover_writable_hdf_sites() + + +def test_no_production_dataframe_to_hdf_sink_bypasses_registry() -> None: + sites: list[str] = [] + for root in PRODUCTION_ROOTS: + for path in root.rglob("*.py"): + if "tests" in path.parts: + continue + tree = ast.parse(path.read_text()) + for node in ast.walk(tree): + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and node.func.attr == "to_hdf" + ): + sites.append( + f"{path.relative_to(REPOSITORY_ROOT).as_posix()}:{node.lineno}" + ) + assert sites == [] diff --git a/packages/microcosm-build/tests/test_logbook.py b/packages/microcosm-build/tests/test_logbook.py index 49312b68..d46752e0 100644 --- a/packages/microcosm-build/tests/test_logbook.py +++ b/packages/microcosm-build/tests/test_logbook.py @@ -93,6 +93,11 @@ def test_sql_schema_round_trip_matches_python_hash_surface() -> None: assert "ALTER EXTENSION pgcrypto SET SCHEMA extensions" in sql assert "trim_scale((p_value #>> '{}')::numeric)::text" in sql assert "rung IN ('f001', 'f010', 'f100')" in sql + rung_migration = ( + MIGRATION.parent / "20260813000000_logbook_f004_rung.sql" + ).read_text(encoding="utf-8") + assert "rung IN ('f001', 'f004', 'f010', 'f100')" in rung_migration + assert "builds_rung_fraction_token" in rung_migration assert "CHECK (logbook.valid_build_phases(phases_reached))" in builds assert "CHECK (logbook.valid_gate_verdicts(gate_verdicts))" in builds assert "phases_reached jsonb NOT NULL DEFAULT" not in builds @@ -245,7 +250,7 @@ def test_published_row_requires_an_artifact_location() -> None: LogbookRow.create(**_row_kwargs(disposition="published")) -@pytest.mark.parametrize("rung", ["f001", "f010", "f100"]) +@pytest.mark.parametrize("rung", ["f001", "f004", "f010", "f100"]) def test_standard_scale_rungs_are_accepted(rung: str) -> None: assert LogbookRow.create(**_row_kwargs(rung=rung)).rung == rung diff --git a/packages/microcosm-build/tests/test_uk_stage_checkpoints.py b/packages/microcosm-build/tests/test_uk_stage_checkpoints.py index a3aadc3d..f71c5da5 100644 --- a/packages/microcosm-build/tests/test_uk_stage_checkpoints.py +++ b/packages/microcosm-build/tests/test_uk_stage_checkpoints.py @@ -8,6 +8,7 @@ from __future__ import annotations +import hashlib from pathlib import Path import pandas as pd @@ -84,6 +85,22 @@ def test_round_trip_restores_metadata_and_content(tmp_path: Path) -> None: assert uk_time_period(predecessor.frame) == "2023" +def test_uk_no_extension_checkpoint_keeps_its_schema_2_byte_golden( + tmp_path: Path, +) -> None: + frame = _frame() + completed = _runtime(tmp_path).complete( + "retain", + frame, + metadata=uk_stage_metadata(frame), + ) + + assert completed.path.name == "000_retain.frame.h5" + assert hashlib.sha256(completed.path.read_bytes()).hexdigest() == ( + "7fd5d25833f395b9eac57fcb0bc6537a344862a024ee981daf167949722a17ee" + ) + + def test_checkpoint_without_frame_metadata_fails_closed(tmp_path: Path) -> None: frame = _frame() runtime = _runtime(tmp_path) diff --git a/packages/microcosm-build/tests/test_us_acs_multispine_base_builder.py b/packages/microcosm-build/tests/test_us_acs_multispine_base_builder.py index 3ac894d1..08fd3aec 100644 --- a/packages/microcosm-build/tests/test_us_acs_multispine_base_builder.py +++ b/packages/microcosm-build/tests/test_us_acs_multispine_base_builder.py @@ -8,7 +8,6 @@ import subprocess import sys from pathlib import Path -from types import ModuleType import numpy as np import pandas as pd @@ -20,7 +19,13 @@ load_legacy_calibrated_us_h5, write_nullable_us_h5, ) -from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights +from microcosm.frame import ( + US_SCHEMA, + Frame, + WeightKind, + Weights, + put_frame_table, +) def _shim_path() -> Path: @@ -170,33 +175,24 @@ def test_shim_preserves_legacy_write_signature_and_default( def test_legacy_loader_and_shim_keep_calibrated_weight_contract( - monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, ) -> None: + # The loader reads legacy artifacts directly via HDFStore — no + # policyengine_us dependency remains — so the contract is exercised + # against a real legacy-shaped file. + pytest.importorskip("tables") source = _frame(weight_kind=WeightKind.DESIGN) - captured: dict[str, str] = {} - - class FakeDataset: - def __init__(self, *, file_path: str) -> None: - captured["file_path"] = file_path - for entity in source.entities: - table = source.table(entity).copy() - if entity == "household": - table["household_weight"] = [7.0, 11.0] - setattr(self, entity, table) - - package = ModuleType("policyengine_us") - package.__path__ = [] # type: ignore[attr-defined] - data = ModuleType("policyengine_us.data") - data.USSingleYearDataset = FakeDataset # type: ignore[attr-defined] - package.data = data # type: ignore[attr-defined] - monkeypatch.setitem(sys.modules, "policyengine_us", package) - monkeypatch.setitem(sys.modules, "policyengine_us.data", data) - - path = Path("legacy.h5") + path = tmp_path / "legacy.h5" + with pd.HDFStore(path, mode="w") as store: + for entity in source.entities: + table = source.table(entity).copy() + if entity == "household": + table["household_weight"] = [7.0, 11.0] + put_frame_table(store, entity, table, preferred_format="fixed") + direct = load_legacy_calibrated_us_h5(path) via_shim = _load_shim_module()._load_base_frame(path) - assert captured["file_path"] == str(path) for loaded in (direct, via_shim): assert loaded.weights_for("household").kind is WeightKind.CALIBRATED assert loaded.weights_for("household").values.tolist() == [7.0, 11.0] diff --git a/packages/microcosm-build/tests/test_us_acs_transfer.py b/packages/microcosm-build/tests/test_us_acs_transfer.py index 0fefeb3f..ff4c7fc6 100644 --- a/packages/microcosm-build/tests/test_us_acs_transfer.py +++ b/packages/microcosm-build/tests/test_us_acs_transfer.py @@ -1236,10 +1236,10 @@ def test_engine_boolean_metadata_restores_primary_qrf_float_h5_donor( PolicyEngineUSVariableMetadataIndex() except ImportError: pytest.skip("requires the policyengine-us [us] extra") - # Primary PUF finalization physically stores QBI boolean-count outputs as - # floats; the supported legacy ACS builder can then load them from HDF as - # its transfer donor. Pin that real producer/HDF representation rather - # than using an unrelated model-required boolean. + # Retiring primary-PUF artifacts physically stored QBI boolean-count + # outputs as floats, and the supported legacy ACS builder can still load + # them from HDF as its transfer donor. Pin that compatibility representation + # rather than using an unrelated model-required boolean. source = tmp_path / "legacy-boolean-donor.h5" pd.DataFrame({"business_is_sstb": [1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0]}).to_hdf( source, key="person", format="fixed" @@ -2071,6 +2071,31 @@ def test_schedule_d_post_transfer_fills_only_newly_imputed_rows( ) assert derived.imputed_recipient_rows == 2 + targets = ( + "long_term_capital_gains_before_response", + "non_sch_d_capital_gains", + ) + bound_contract = acs_transfer_module.acs_transfer_execution_contract_identity( + targets=targets, + derive_schedule_d=False, + ) + suppressed = transfer_acs_inputs( + recipient, + donor, + target_families={"person": {"capital_gain_details": targets}}, + n_estimators=1, + derive_schedule_d=False, + execution_contract=bound_contract, + ) + pd.testing.assert_series_equal( + suppressed.frame.person["schedule_d_capital_gain_distributions"], + cgd_before, + ) + assert all( + item.column != "schedule_d_capital_gain_distributions" + for item in suppressed.imputed_inputs + ) + def test_adult_care_reconciliation_changes_only_imputed_expenses( monkeypatch: pytest.MonkeyPatch, diff --git a/packages/microcosm-build/tests/test_us_adult_care.py b/packages/microcosm-build/tests/test_us_adult_care.py index d8357c16..d9f4fca3 100644 --- a/packages/microcosm-build/tests/test_us_adult_care.py +++ b/packages/microcosm-build/tests/test_us_adult_care.py @@ -259,8 +259,7 @@ def source_peer(*, spine: str) -> Frame: assert set(fit_person["person_support_clone_index"]) == {1} flag = fit_person[_FLAG] - assert flag.dtype == object - assert pd.api.types.infer_dtype(flag, skipna=True) == "boolean" + assert flag.dtype == pd.BooleanDtype() source_channel = fit_person["person_support_channel"] assert flag.loc[source_channel.eq("asec")].notna().all() assert flag.loc[source_channel.eq("acs")].isna().all() diff --git a/packages/microcosm-build/tests/test_us_education_inputs.py b/packages/microcosm-build/tests/test_us_education_inputs.py index 10d8cf9a..7a6124b0 100644 --- a/packages/microcosm-build/tests/test_us_education_inputs.py +++ b/packages/microcosm-build/tests/test_us_education_inputs.py @@ -33,6 +33,9 @@ us_education_inputs_summary, with_us_education_inputs, ) +from microcosm.build.us_runtime.education_inputs import ( + US_EDUCATION_INPUTS_OWNED_OUTPUT_COLUMNS, +) from microcosm.build.us_runtime.source_runtime import us_source_operation_handlers from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights @@ -50,6 +53,10 @@ "educational_assistance", *_EXPECTED_AOTC_COLUMNS, ) +_EXPECTED_OWNED_OUTPUT_COLUMNS = ( + "educational_assistance", + *_EXPECTED_AOTC_COLUMNS, +) def _person_table(rows: list[dict]) -> pd.DataFrame: @@ -136,8 +143,10 @@ def test_stage_declares_the_complete_seven_column_family(self) -> None: assert US_AOTC_ELIGIBILITY_OUTPUT_COLUMNS == _EXPECTED_AOTC_COLUMNS assert US_EDUCATION_INPUTS_OUTPUT_COLUMNS == _EXPECTED_OUTPUT_COLUMNS assert ( - US_EDUCATION_INPUTS_NONCONSTANT_PERSON_COLUMNS - == _EXPECTED_OUTPUT_COLUMNS + US_EDUCATION_INPUTS_OWNED_OUTPUT_COLUMNS == _EXPECTED_OWNED_OUTPUT_COLUMNS + ) + assert ( + US_EDUCATION_INPUTS_NONCONSTANT_PERSON_COLUMNS == _EXPECTED_OUTPUT_COLUMNS ) assert tuple(spec.outputs) == _EXPECTED_OUTPUT_COLUMNS assert US_EDUCATION_INPUTS_REQUIRED_SOURCE_COLUMNS == ( @@ -187,25 +196,69 @@ def test_preserves_upstream_tuition_and_flags_exactly_positive_tuition( for column in _EXPECTED_AOTC_COLUMNS: assert result[column].tolist() == expected + def test_preserves_upstream_tuition_bytes_and_dtype(self) -> None: + tuition = np.asarray([0.0, 1_250.25, 4_000.5], dtype=np.float32) + table = pd.DataFrame( + { + "person_id": [1, 2, 3], + "ED_VAL": [0.0, 0.0, 0.0], + "qualified_tuition_expenses": tuition, + } + ) + expected_bytes = table["qualified_tuition_expenses"].to_numpy().tobytes() + + result = self._derive(table) + + assert result["qualified_tuition_expenses"].dtype == np.dtype("float32") + assert ( + result["qualified_tuition_expenses"].to_numpy().tobytes() == expected_bytes + ) + def test_maps_ed_val_to_educational_assistance(self) -> None: result = self._derive(_person_table([{"ED_VAL": 750.0}, {"ED_VAL": 0.0}])) assert result["educational_assistance"].tolist() == [750.0, 0.0] - def test_amounts_are_numeric_finite_and_clipped_nonnegative(self) -> None: + def test_numeric_strings_are_validated_without_rewriting_tuition(self) -> None: result = self._derive( _person_table( [ - {"ED_VAL": -10.0, "qualified_tuition_expenses": -20.0}, - {"ED_VAL": np.nan, "qualified_tuition_expenses": np.nan}, {"ED_VAL": "300", "qualified_tuition_expenses": "1200"}, ] ) ) - assert result["educational_assistance"].tolist() == [0.0, 0.0, 300.0] - assert result["qualified_tuition_expenses"].tolist() == [0.0, 0.0, 1_200.0] + assert result["educational_assistance"].tolist() == [300.0] + assert result["qualified_tuition_expenses"].tolist() == ["1200"] for column in _EXPECTED_AOTC_COLUMNS: - assert result[column].tolist() == [False, False, True] + assert result[column].tolist() == [True] + + @pytest.mark.parametrize( + ("column", "value"), + [ + ("ED_VAL", np.nan), + ("ED_VAL", np.inf), + ("ED_VAL", "not-a-number"), + ("qualified_tuition_expenses", np.nan), + ("qualified_tuition_expenses", np.inf), + ("qualified_tuition_expenses", "not-a-number"), + ], + ) + def test_nonnumeric_or_nonfinite_amount_is_refused( + self, column: str, value: object + ) -> None: + with pytest.raises( + SourceRuntimeError, + match=rf"{column!r} contains 1 nonnumeric or nonfinite value", + ): + self._derive(_person_table([{column: value}])) + + @pytest.mark.parametrize("column", ["ED_VAL", "qualified_tuition_expenses"]) + def test_negative_amount_is_refused(self, column: str) -> None: + with pytest.raises( + SourceRuntimeError, + match=rf"{column!r} contains 1 negative value", + ): + self._derive(_person_table([{column: -1.0}])) @pytest.mark.parametrize("missing", US_EDUCATION_INPUTS_REQUIRED_SOURCE_COLUMNS) def test_missing_source_column_is_named(self, missing: str) -> None: @@ -256,6 +309,29 @@ def test_with_inputs_writes_the_complete_family(self) -> None: for column in _EXPECTED_AOTC_COLUMNS: assert person[column].tolist() == [False, True] + def test_with_inputs_does_not_rewrite_upstream_tuition(self) -> None: + frame = _us_frame( + [ + {"ED_VAL": 600.0, "qualified_tuition_expenses": 0.0}, + {"ED_VAL": 0.0, "qualified_tuition_expenses": 2_000.25}, + ] + ) + frame.table("person")["qualified_tuition_expenses"] = frame.table("person")[ + "qualified_tuition_expenses" + ].astype(np.float32) + expected = frame.table("person")["qualified_tuition_expenses"] + expected_bytes = expected.to_numpy().tobytes() + + result = with_us_education_inputs( + frame, + seed=0, + time_period=TIME_PERIOD, + ) + actual = result.table("person")["qualified_tuition_expenses"] + + assert actual.dtype == expected.dtype == np.dtype("float32") + assert actual.to_numpy().tobytes() == expected_bytes + def test_frame_with_coherent_signal_passes_through_untouched(self) -> None: derived = with_us_education_inputs( _us_frame(_plausible_rows()), seed=0, time_period=TIME_PERIOD @@ -332,10 +408,14 @@ def test_puf_support_to_education_stage_preserves_real_source_signal( channel == BASE_ASEC_SUPPORT_CHANNEL, "qualified_tuition_expenses", ].any() - assert person.loc[ - channel == PUF_TAX_DETAIL_SUPPORT_CHANNEL, - "qualified_tuition_expenses", - ].gt(0).any() + assert ( + person.loc[ + channel == PUF_TAX_DETAIL_SUPPORT_CHANNEL, + "qualified_tuition_expenses", + ] + .gt(0) + .any() + ) result = with_us_education_inputs( imputed, diff --git a/packages/microcosm-build/tests/test_us_exact_k_ladder_e2e.py b/packages/microcosm-build/tests/test_us_exact_k_ladder_e2e.py index fcfd7c06..c071a4f5 100644 --- a/packages/microcosm-build/tests/test_us_exact_k_ladder_e2e.py +++ b/packages/microcosm-build/tests/test_us_exact_k_ladder_e2e.py @@ -15,13 +15,17 @@ US_MULTISPINE_AGREEMENT_DIAGNOSTICS_ARTIFACT_KIND, US_MULTISPINE_POOL_H5_ARTIFACT_KIND, US_MULTISPINE_POOL_MANIFEST_ARTIFACT_KIND, - US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION, load_simulation_ready_us_multispine_pool, write_nullable_us_h5, ) from microcosm.calibrate import TargetRegistry, TargetSpec from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights +# This minimal downstream-consumer fixture intentionally models the preserved +# pre-stacked publication envelope, not a schema-6 stacked artifact without its +# required DAG authority fields. +_LEGACY_POOL_MANIFEST_SCHEMA_VERSION = 4 + def _builder_module(): root = Path(__file__).resolve().parents[3] @@ -98,7 +102,7 @@ def _write_ready_pool(tmp_path: Path) -> Path: json.dumps( { "artifact_kind": (US_MULTISPINE_AGREEMENT_DIAGNOSTICS_ARTIFACT_KIND), - "schema_version": US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION, + "schema_version": _LEGACY_POOL_MANIFEST_SCHEMA_VERSION, "simulation_ready": True, "publication_run_id": run_id, "agreement_gate": agreement_gate, @@ -110,17 +114,36 @@ def _write_ready_pool(tmp_path: Path) -> Path: json.dumps( { "artifact_kind": US_MULTISPINE_POOL_MANIFEST_ARTIFACT_KIND, - "schema_version": US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION, + "schema_version": _LEGACY_POOL_MANIFEST_SCHEMA_VERSION, "status": "simulation_ready", "simulation_ready": True, "publication_run_id": run_id, "period": 2024, + "operator_order": [ + "assemble", + "clone", + "impute", + "derive", + "seed", + "simulate", + "agreement", + ], + "stage_receipts": { + stage: {"operator": stage} + for stage in ("impute", "derive", "seed", "simulate") + }, "stage_checkpoints": { + "artifact_kind": ( + "populace_us_multispine_pool_checkpoint_provenance" + ), + "schema_version": 1, + "materializer_version": 3, + "enabled": False, "agreement": { "source": "always_fresh", "cached": False, "terminal_verdict_persisted": False, - } + }, }, "agreement_gate": agreement_gate, "provenance_counts": {"household": {"rows": 8}}, diff --git a/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py b/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py index 0b6bc629..1c4c4a62 100644 --- a/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py +++ b/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py @@ -456,9 +456,10 @@ def test__given_target_frame_checkpoint__then_builder_round_trips_frame( ssi_take_up_assignment_sha256="ssi-flags-sha", selection_identities_sha256=None, ) - # 10 = #557 preserves the staged retirement surface through release - # materialization; pre-#557 QRF-refitted checkpoints (9) must not serve. - assert identity["materializer_version"] == 10 + # 11 = target checkpoints preserve nullable booleans explicitly; schema 2 + # distinguishes the new values+mask codec from schema-1 checkpoints. + assert identity["schema_version"] == 2 + assert identity["materializer_version"] == 11 # The SSI prior-weight basis is identity-bearing (microcosm#543 instance # 2): unflagged runs carry the key as None. assert identity["ssi_take_up_prior_weight_basis_sha256"] is None @@ -509,6 +510,160 @@ def test__given_target_frame_checkpoint__then_builder_round_trips_frame( ) +def test_target_frame_checkpoint_nullable_boolean_storage_is_explicit( + monkeypatch, + tmp_path, + small_frame, +) -> None: + builder = _load_builder_module() + monkeypatch.setattr(builder, "US_SCHEMA", small_frame.schema) + tables = { + entity: small_frame.table(entity).copy() for entity in small_frame.entities + } + tables["person"]["native_boolean"] = np.asarray( + [False, True, False, True], dtype=np.bool_ + ) + tables["person"]["complete_nullable_boolean"] = pd.Series( + [True, False, True, False], dtype="boolean" + ) + tables["person"]["missing_nullable_boolean"] = pd.Series( + [True, pd.NA, False, pd.NA], dtype="boolean" + ) + frame = Frame( + tables, + small_frame.schema, + {"household": small_frame.weights_for("household")}, + small_frame.strata, + ) + identity = { + "schema_version": builder.TARGET_FRAME_CHECKPOINT_SCHEMA_VERSION, + "materializer_version": builder.TARGET_FRAME_CHECKPOINT_MATERIALIZER_VERSION, + } + path = tmp_path / "nullable-booleans.h5" + + builder._write_target_frame_checkpoint( + path, + frame=frame, + identity=identity, + compilation={}, + ) + loaded = builder._read_target_frame_checkpoint( + path, + identity=identity, + target_specs=(), + ) + + assert loaded is not None + observed = loaded[0].table("person") + assert observed["native_boolean"].dtype == np.dtype(np.bool_) + assert observed["complete_nullable_boolean"].dtype == pd.BooleanDtype() + assert observed["missing_nullable_boolean"].dtype == pd.BooleanDtype() + pd.testing.assert_series_equal( + observed["missing_nullable_boolean"], + tables["person"]["missing_nullable_boolean"], + ) + with h5py.File(path, mode="r") as h5: + person = h5["tables"]["person"] + columns = json.loads(str(person.attrs["columns_json"])) + groups = { + column: person["columns"][f"{columns.index(column):05d}"] + for column in ( + "native_boolean", + "complete_nullable_boolean", + "missing_nullable_boolean", + ) + } + assert bool(groups["native_boolean"].attrs["nullable"]) is False + assert bool(groups["complete_nullable_boolean"].attrs["nullable"]) is True + assert bool(groups["complete_nullable_boolean"].attrs["has_null_mask"]) is False + missing = groups["missing_nullable_boolean"] + assert bool(missing.attrs["nullable"]) is True + assert bool(missing.attrs["has_null_mask"]) is True + assert ( + np.asarray(missing["values"]).tobytes() + == np.asarray([True, False, False, False], dtype=np.bool_).tobytes() + ) + assert np.asarray(missing["null_mask"]).dtype == np.dtype(np.uint8) + np.testing.assert_array_equal( + np.asarray(missing["null_mask"]), + np.asarray([0, 1, 0, 1], dtype=np.uint8), + ) + + +@pytest.mark.parametrize( + "corrupt", + ( + "missing_mask", + "noncanonical_hidden_bit", + "nonbinary_mask", + "unexpected_mask", + "missing_metadata", + ), +) +def test_target_frame_checkpoint_rejects_malformed_boolean_storage( + tmp_path, + small_frame, + corrupt, +) -> None: + builder = _load_builder_module() + tables = { + entity: small_frame.table(entity).copy() for entity in small_frame.entities + } + tables["person"]["flag"] = pd.Series([True, pd.NA, False, pd.NA], dtype="boolean") + frame = Frame( + tables, + small_frame.schema, + {"household": small_frame.weights_for("household")}, + small_frame.strata, + ) + path = tmp_path / f"malformed-{corrupt}.h5" + builder._write_target_frame_checkpoint( + path, + frame=frame, + identity={}, + compilation={}, + ) + + with h5py.File(path, mode="r+") as h5: + person = h5["tables"]["person"] + columns = json.loads(str(person.attrs["columns_json"])) + group = person["columns"][f"{columns.index('flag'):05d}"] + if corrupt == "missing_mask": + del group["null_mask"] + elif corrupt == "noncanonical_hidden_bit": + group["values"][1] = True + elif corrupt == "nonbinary_mask": + group["null_mask"][1] = 2 + elif corrupt == "unexpected_mask": + group.attrs["has_null_mask"] = False + else: + del group.attrs["nullable"] + + with h5py.File(path, mode="r") as h5, pytest.raises(RuntimeError): + builder._read_checkpoint_dataframe(h5["tables"]["person"]) + + +def test_target_frame_checkpoint_rejects_schema_one( + monkeypatch, + tmp_path, + small_frame, +) -> None: + builder = _load_builder_module() + monkeypatch.setattr(builder, "US_SCHEMA", small_frame.schema) + path = tmp_path / "schema-one.h5" + builder._write_target_frame_checkpoint( + path, + frame=small_frame, + identity={}, + compilation={}, + ) + with h5py.File(path, mode="r+") as h5: + h5.attrs["schema_version"] = 1 + + with pytest.raises(RuntimeError, match="schema mismatch.*got 1, expected 2"): + builder._read_target_frame_checkpoint(path, identity={}, target_specs=()) + + def test__given_stale_materializer_version_checkpoint__then_builder_rejects_it( monkeypatch, tmp_path, @@ -516,10 +671,9 @@ def test__given_stale_materializer_version_checkpoint__then_builder_rejects_it( ) -> None: """A checkpoint stored under a superseded materializer version must not load. - #557 changed the staged retirement-surface semantics: version-9 - checkpoints can carry release-refitted leaves instead of the preserved - support-built surface. The version constant participates in the identity - comparison; this pins the stored-9 versus current-10 rejection directly. + Version 11 adds the lossless nullable-boolean checkpoint codec. The version + constant participates in the identity comparison; this pins stored-10 + versus current-11 rejection directly. """ builder = _load_builder_module() monkeypatch.setattr(builder, "US_SCHEMA", small_frame.schema) @@ -553,10 +707,10 @@ def test__given_stale_materializer_version_checkpoint__then_builder_rejects_it( ssi_take_up_assignment_sha256="ssi-flags-sha", selection_identities_sha256=None, ) - # 9 = the pre-#557 release-refit world; 8 = the still-older pre-#374 blend - # world. Both must miss against expected version 10. - stale_identity = {**dict(identity), "materializer_version": 9} - older_identity = {**dict(identity), "materializer_version": 8} + # 10 = the pre-nullable-boolean-codec world; 9 = the still-older pre-#557 + # release-refit world. Both must miss against expected version 11. + stale_identity = {**dict(identity), "materializer_version": 10} + older_identity = {**dict(identity), "materializer_version": 9} path = tmp_path / "target_frame_checkpoint.h5" builder._write_target_frame_checkpoint( path, diff --git a/packages/microcosm-build/tests/test_us_late_producer_dag.py b/packages/microcosm-build/tests/test_us_late_producer_dag.py new file mode 100644 index 00000000..aa83a6f7 --- /dev/null +++ b/packages/microcosm-build/tests/test_us_late_producer_dag.py @@ -0,0 +1,1371 @@ +"""Late-stage producer DAG doctrine regressions (microcosm#653).""" + +from __future__ import annotations + +from collections import Counter, OrderedDict + +import pytest + +import microcosm.build.us_runtime.acs_pums as acs_pums_module +import microcosm.build.us_runtime.acs_transfer as acs_transfer_module +from microcosm.build.us_runtime.acs_income_universe import ( + ACS_PUMS_EARNINGS_SOURCE_COLUMNS, +) +from microcosm.build.us_runtime.education_inputs import ( + US_EDUCATION_INPUTS_OUTPUT_COLUMNS, + US_EDUCATION_INPUTS_OWNED_OUTPUT_COLUMNS, +) +from microcosm.build.us_runtime.late_producer_dag import ( + ProducerContract, + ProducerInput, + ProducerInputColumn, + ProducerOutput, + derive_producer_schedule, + run_producer_when_ready, +) +from microcosm.build.us_runtime.multispine_pool import ( + POOL_OPERATOR_CONTRACTS, + POOL_POST_CLONE_SOURCE_OPERATOR_ORDER, +) +from microcosm.build.us_runtime.operator_boundary import ( + FORMULA_OWNED_SOURCE_COLUMNS, + PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES, +) +from microcosm.build.us_runtime.puf_capital_gains_tail import ( + PUF_CAPITAL_GAINS_TAIL_PERSON_COLUMNS, + PUF_CAPITAL_GAINS_TAIL_TAX_UNIT_COLUMNS, +) +from microcosm.build.us_runtime.puf_support import ( + PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS, + PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS, +) +from microcosm.build.us_runtime.us_late_overlap_ownership import ( + US_LATE_OVERLAP_OWNERSHIP_TARGETS, + US_LATE_SOURCE_CALLBACK_PASSTHROUGH_OUTPUTS, + us_late_overlap_ownership_receipt, + validate_us_late_overlap_ownership_receipt, +) +from microcosm.build.us_runtime.us_late_producer_registry import ( + CANONICAL_US_LATE_PRODUCER_REGISTRY, + CANONICAL_US_LATE_PRODUCER_SCHEDULE, + CANONICAL_US_LATE_TRANSFER_GROUPS, + US_LATE_ACS_EARNINGS_UNIVERSE_INPUT_INVENTORY, + US_LATE_ACS_EARNINGS_UNIVERSE_STAGE, + US_LATE_EXTERNAL_STAGES, + US_LATE_PRIMARY_PUF_INPUT_INVENTORY, + US_LATE_PRIMARY_PUF_STAGE, + US_LATE_SOURCE_FINALIZER_STAGE, + US_LATE_SOURCE_INPUT_INVENTORIES, + US_LATE_TRANSFER_INPUT_INVENTORIES, + source_producer_name, + transfer_producer_name, + us_late_producer_schedule_receipt, +) + + +def _contract(name: str, *dependencies: str) -> ProducerContract: + return ProducerContract( + name=name, + kind="fixture", + inputs=tuple( + ProducerInput( + entity="person", + column=f"{dependency}_output", + required_scope="whole_pool", + producing_stage=dependency, + ) + for dependency in dependencies + ), + outputs=( + ProducerOutput( + entity="person", + column=f"{name}_output", + coverage_scope="whole_pool", + ), + ), + ) + + +def test_unfilled_late_input_refuses_before_producer_runs() -> None: + requirement = ProducerInput( + entity="person", + column="late_input", + required_scope="cps_projection", + producing_stage="transfer:person/puf_tax_itemization__batch_5", + ) + consumer = ProducerContract( + name="with_fixture_consumer", + kind="source_operator", + inputs=(requirement,), + outputs=(), + ) + invoked = False + + def callback() -> None: + nonlocal invoked + invoked = True + + with pytest.raises( + ValueError, + match=( + r"(?s)with_fixture_consumer.*person\.late_input.*1 unfilled.*" + r"cps_projection.*transfer:person/puf_tax_itemization__batch_5" + ), + ): + run_producer_when_ready( + consumer, + callback, + unfilled_rows={requirement: 1}, + invalid_rows={requirement: 0}, + absence_receipts={}, + ) + + assert invoked is False + + +def test_declared_absence_never_tolerates_invalid_input() -> None: + receipt_id = "optional_input:consumer:predictor" + requirement = ProducerInput( + entity="person", + column="@effective:predictor", + required_scope="whole_pool", + producing_stage="post_clone_input_surface", + tolerated_absence_receipts=(receipt_id,), + ) + consumer = ProducerContract( + name="consumer", + kind="fixture", + inputs=(requirement,), + outputs=(), + ) + forged_absence = { + receipt_id: { + "receipt_id": receipt_id, + "status": "declared_absence", + "entity": "person", + "column": "@effective:predictor", + "required_scope": "whole_pool", + "rows": 1, + } + } + + with pytest.raises( + ValueError, + match=( + r"(?s)consumer.*person\.@effective:predictor.*1 invalid.*" + r"post_clone_input_surface" + ), + ): + run_producer_when_ready( + consumer, + lambda: pytest.fail("invalid input reached callback"), + unfilled_rows={requirement: 0}, + invalid_rows={requirement: 1}, + absence_receipts=forged_absence, + ) + + +def test_declared_absence_rejects_a_cross_producer_receipt() -> None: + receipt_id = "optional_input:consumer:predictor" + requirement = ProducerInput( + entity="person", + column="@effective:predictor", + required_scope="whole_pool", + producing_stage="post_clone_input_surface", + tolerated_absence_receipts=(receipt_id,), + ) + consumer = ProducerContract("consumer", "fixture", (requirement,), ()) + wrong_owner = { + receipt_id: { + "receipt_id": receipt_id, + "status": "declared_absence", + "entity": "person", + "column": "@effective:predictor", + "required_scope": "whole_pool", + "rows": 1, + "producer": "different_consumer", + "reason": "optional availability-pattern input", + } + } + + with pytest.raises( + ValueError, + match=r"(?s)consumer.*@effective:predictor.*1 unfilled", + ): + run_producer_when_ready( + consumer, + lambda: pytest.fail("cross-producer absence reached callback"), + unfilled_rows={requirement: 1}, + invalid_rows={requirement: 0}, + absence_receipts=wrong_owner, + ) + + +def test_readiness_requires_exact_declared_count_surfaces() -> None: + requirement = ProducerInput( + entity="person", + column="required_input", + required_scope="whole_pool", + producing_stage="producer", + ) + consumer = ProducerContract("consumer", "fixture", (requirement,), ()) + + with pytest.raises( + ValueError, + match=r"consumer.*unfilled-row readiness.*missing=.*required_input", + ): + run_producer_when_ready( + consumer, + lambda: pytest.fail("omitted input reached callback"), + unfilled_rows={}, + invalid_rows={requirement: 0}, + absence_receipts={}, + ) + + with pytest.raises( + ValueError, + match=r"consumer.*invalid-value readiness.*missing=.*required_input", + ): + run_producer_when_ready( + consumer, + lambda: pytest.fail("omitted input reached callback"), + unfilled_rows={requirement: 0}, + invalid_rows={}, + absence_receipts={}, + ) + + +def test_synthetic_producer_cycle_is_rejected_with_named_cycle() -> None: + registry = { + "alpha": _contract("alpha", "charlie"), + "bravo": _contract("bravo", "alpha"), + "charlie": _contract("charlie", "bravo"), + } + + with pytest.raises( + RuntimeError, + match=r"alpha -> bravo -> charlie -> alpha", + ): + derive_producer_schedule(registry) + + +def _scoped_dependency_registry( + *, + output_scope: str, + required_scope: str, +) -> dict[str, ProducerContract]: + return { + "producer": ProducerContract( + "producer", + "fixture", + (), + (ProducerOutput("person", "shared", output_scope),), + ), + "consumer": ProducerContract( + "consumer", + "fixture", + ( + ProducerInput( + "person", + "shared", + required_scope, + "producer", + ), + ), + (), + ), + } + + +def test_schedule_rejects_producer_output_with_insufficient_scope() -> None: + registry = _scoped_dependency_registry( + output_scope="asec_source", + required_scope="whole_pool", + ) + + with pytest.raises( + ValueError, + match=( + r"(?s)scope_mismatches=.*consumer.*producer.*person\.shared.*" + r"whole_pool.*asec_source" + ), + ): + derive_producer_schedule(registry) + + +@pytest.mark.parametrize( + ("output_scope", "required_scope"), + ( + ("whole_pool", "whole_pool"), + ("whole_pool", "asec_source"), + ("whole_pool", "puf_clone"), + ("asec_source", "asec_source"), + ("puf_clone", "puf_clone"), + ("receipt", "whole_pool"), + ), +) +def test_schedule_accepts_declared_scope_coverage( + output_scope: str, + required_scope: str, +) -> None: + schedule = derive_producer_schedule( + _scoped_dependency_registry( + output_scope=output_scope, + required_scope=required_scope, + ) + ) + + assert schedule.edges == (("producer", "consumer"),) + assert schedule.waves == (("producer",), ("consumer",)) + + +def test_derived_schedule_is_byte_stable_under_registry_iteration_order() -> None: + contracts = ( + _contract("alpha"), + _contract("bravo"), + _contract("charlie", "alpha", "bravo"), + _contract("delta", "charlie"), + ) + forward = OrderedDict((contract.name, contract) for contract in contracts) + reverse = OrderedDict((contract.name, contract) for contract in reversed(contracts)) + + forward_schedule = derive_producer_schedule(forward) + reverse_schedule = derive_producer_schedule(reverse) + + assert forward_schedule.order == reverse_schedule.order + assert forward_schedule.waves == reverse_schedule.waves + assert forward_schedule.edges == reverse_schedule.edges + assert forward_schedule.canonical_json == reverse_schedule.canonical_json + assert forward_schedule.sha256 == reverse_schedule.sha256 + + +def test_canonical_us_late_registry_has_exact_producer_surface() -> None: + registry = CANONICAL_US_LATE_PRODUCER_REGISTRY + groups = CANONICAL_US_LATE_TRANSFER_GROUPS + + assert len(registry) == 38 + assert len(groups) == 19 + assert sum(len(group.targets) for group in groups) == 70 + assert {contract.kind for contract in registry.values()} == { + "primary_puf", + "acs_earnings_universe", + "post_clone_source", + "late_transfer", + "source_finalizer", + } + assert len(registry[US_LATE_PRIMARY_PUF_STAGE].inputs) == 119 + primary_outputs = registry[US_LATE_PRIMARY_PUF_STAGE].outputs + assert len(primary_outputs) == 100 + assert sum(output.coverage_scope == "puf_clone" for output in primary_outputs) == 64 + assert ( + sum(output.coverage_scope == "whole_pool" for output in primary_outputs) == 35 + ) + assert sum(output.coverage_scope == "acs_source" for output in primary_outputs) == 1 + assert { + (output.entity, output.column, output.coverage_scope) + for output in primary_outputs + if output.coverage_scope == "acs_source" + } == { + ("household", "TYPEHUGQ", "acs_source"), + } + assert { + (output.entity, output.column, output.coverage_scope) + for output in primary_outputs + if output.coverage_scope == "whole_pool" + } >= { + ("person", "person_support_clone_index", "whole_pool"), + ("person", "s_corp_income", "whole_pool"), + ("frame", "@us_puf_clone_attachment_manifest", "whole_pool"), + } + assert all(contract.inputs for contract in registry.values()) + assert { + name + for name, contract in registry.items() + if contract.kind == "post_clone_source" + } == { + source_producer_name(operator) + for operator in POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + } + assert { + name for name, contract in registry.items() if contract.kind == "late_transfer" + } == {group.name for group in groups} + for group in groups: + assert { + (output.entity, output.column) + for output in CANONICAL_US_LATE_PRODUCER_REGISTRY[group.name].outputs + } == {(group.entity, target) for target in group.targets} + + +def test_late_overlap_ownership_exhausts_every_permitted_dual_write() -> None: + primary = { + (entity, column) + for entity, columns in PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES[ + "primary_puf_qrf" + ].items() + for column in columns + } + source_writes = { + (entity, column) + for operator in POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + for entity, columns in PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES[ + POOL_OPERATOR_CONTRACTS[operator].family + ].items() + for column in columns + if column not in FORMULA_OWNED_SOURCE_COLUMNS.get(entity, ()) + } + callback_passthroughs = { + target + for targets in US_LATE_SOURCE_CALLBACK_PASSTHROUGH_OUTPUTS.values() + for target in targets + } + source_touches = source_writes | callback_passthroughs + transfer = { + (group.entity, target) + for group in CANONICAL_US_LATE_TRANSFER_GROUPS + for target in group.targets + } + tail_owned = { + *(("person", column) for column in PUF_CAPITAL_GAINS_TAIL_PERSON_COLUMNS), + *(("tax_unit", column) for column in PUF_CAPITAL_GAINS_TAIL_TAX_UNIT_COLUMNS), + } + recipient_owned = { + *(("person", column) for column in PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS), + *(("tax_unit", column) for column in PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS), + } - tail_owned + declared = set(US_LATE_OVERLAP_OWNERSHIP_TARGETS) + + assert len(primary) == 65 + assert len(source_writes) == 35 + assert len(source_writes & transfer) == 29 + assert len(transfer) == 70 + assert len(recipient_owned) == 60 + assert ( + callback_passthroughs + == { + ("person", column) + for column in set(US_EDUCATION_INPUTS_OUTPUT_COLUMNS) + - set(US_EDUCATION_INPUTS_OWNED_OUTPUT_COLUMNS) + } + == {("person", "qualified_tuition_expenses")} + ) + assert primary & source_touches & transfer & recipient_owned == declared + assert primary & source_writes & transfer & recipient_owned == { + ("person", "traditional_ira_contributions_desired"), + ("person", "self_employed_pension_contributions_desired"), + } + assert primary & source_touches & transfer & tail_owned == set() + + receipt = dict(us_late_overlap_ownership_receipt()) + assert validate_us_late_overlap_ownership_receipt(receipt) == receipt["sha256"] + assert len(receipt["targets"]) == 3 + assert len(receipt["ownership"]) == 18 + assert {(row["entity"], row["target"]) for row in receipt["ownership"]} == declared + assert {(row["origin"], row["clone_index"]) for row in receipt["ownership"]} == { + (origin, clone) for origin in ("asec", "acs") for clone in range(3) + } + for row in receipt["ownership"]: + assert sum(action["owns_final"] for action in row["producer_actions"]) == 1 + owner = next( + action["producer"] + for action in row["producer_actions"] + if action["owns_final"] + ) + assert owner == row["final_owner"] + + source_action_by_cell = { + (row["target"], row["origin"], row["clone_index"]): next( + action["action"] + for action in row["producer_actions"] + if action["producer"] == "source:with_us_education_inputs" + ) + for row in receipt["ownership"] + if row["target"] == "qualified_tuition_expenses" + } + for clone_index in range(3): + assert ( + source_action_by_cell[("qualified_tuition_expenses", "asec", clone_index)] + == "consume_only_byte_exact_noop" + ) + assert ( + source_action_by_cell[("qualified_tuition_expenses", "acs", clone_index)] + == "origin_projection_masked_noop" + ) + + owner_by_cell = { + (row["target"], row["origin"], row["clone_index"]): row["final_owner"] + for row in receipt["ownership"] + } + for origin in ("asec", "acs"): + assert owner_by_cell[("qualified_tuition_expenses", origin, 0)] == ( + "transfer:person/puf_tax_itemization__batch_2" + ) + for clone_index in (1, 2): + assert ( + owner_by_cell[("qualified_tuition_expenses", origin, clone_index)] + == US_LATE_PRIMARY_PUF_STAGE + ) + for target in ( + "traditional_ira_contributions_desired", + "self_employed_pension_contributions_desired", + ): + for clone_index in range(3): + assert owner_by_cell[(target, "asec", clone_index)] == source_producer_name( + "with_us_retirement_contribution_inputs" + ) + assert owner_by_cell[(target, "acs", 0)] == transfer_producer_name( + "person", + "puf_tax_itemization__batch_2" + if target == "traditional_ira_contributions_desired" + else "puf_tax_itemization__batch_3", + ) + for clone_index in (1, 2): + assert owner_by_cell[(target, "acs", clone_index)] == ( + US_LATE_PRIMARY_PUF_STAGE + ) + + finalization_by_cell = { + (row["target"], row["origin"], row["clone_index"]): row["finalization"] + for row in receipt["ownership"] + } + for target in ( + "traditional_ira_contributions_desired", + "self_employed_pension_contributions_desired", + ): + assert finalization_by_cell[(target, "asec", 2)] == ( + "byte_exact_clone_1_mirror" + ) + + schedule_receipt = us_late_producer_schedule_receipt() + assert schedule_receipt["overlap_ownership"] == receipt + + +def test_primary_puf_inventory_declares_exact_read_before_write_surface() -> None: + requirements = { + requirement.label: requirement + for requirement in US_LATE_PRIMARY_PUF_INPUT_INVENTORY.requirements + } + + assert len(requirements) == 114 + assert tuple( + (item.entity, item.column, item.value_kind) + for item in requirements["filing_status"].alternatives[0] + ) == (("tax_unit", "filing_status_input", "non_null"),) + assert requirements["age"].alternatives[0][0].value_kind == "finite_numeric" + allocation_basis = { + label.removeprefix("person_output_allocation_basis:") + for label in requirements + if label.startswith("person_output_allocation_basis:") + } + assert allocation_basis == set(PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS) + assert all( + requirements[f"person_output_allocation_basis:{column}"].optional + for column in PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS + ) + tax_unit_passthrough = { + label.removeprefix("tax_unit_output_passthrough:") + for label in requirements + if label.startswith("tax_unit_output_passthrough:") + } + assert tax_unit_passthrough == set(PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS) + assert all( + requirements[f"tax_unit_output_passthrough:{column}"].optional + for column in PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS + ) + assert requirements["qualified_tuition_allocation_fallback"].optional + + primary = CANONICAL_US_LATE_PRODUCER_REGISTRY[US_LATE_PRIMARY_PUF_STAGE] + contract_inputs = {item.column: item for item in primary.inputs} + for column in PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS: + label = f"person_output_allocation_basis:{column}" + requirement = requirements[label] + assert tuple( + (item.entity, item.column, item.value_kind) + for item in requirement.alternatives[0] + ) == (("person", column, "finite_numeric"),) + assert contract_inputs[f"@effective:{label}"].tolerated_absence_receipts == ( + f"optional_input:{US_LATE_PRIMARY_PUF_STAGE}:{label}", + ) + for column in PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS: + label = f"tax_unit_output_passthrough:{column}" + requirement = requirements[label] + assert tuple( + (item.entity, item.column, item.value_kind) + for item in requirement.alternatives[0] + ) == (("tax_unit", column, "finite_numeric"),) + assert contract_inputs[f"@effective:{label}"].tolerated_absence_receipts == ( + f"optional_input:{US_LATE_PRIMARY_PUF_STAGE}:{label}", + ) + fallback = requirements["qualified_tuition_allocation_fallback"] + assert tuple( + (item.entity, item.column, item.value_kind) for item in fallback.alternatives[0] + ) == (("person", "is_full_time_college_student", "finite_numeric"),) + assert contract_inputs[ + "@effective:qualified_tuition_allocation_fallback" + ].tolerated_absence_receipts == ( + f"optional_input:{US_LATE_PRIMARY_PUF_STAGE}:" + "qualified_tuition_allocation_fallback", + ) + typehugq = requirements["validated_structure:TYPEHUGQ"] + assert typehugq.required_scope == "acs_source" + declared_typehugq = contract_inputs["@effective:validated_structure:TYPEHUGQ"] + assert declared_typehugq.required_scope == "acs_source" + assert declared_typehugq.tolerated_absence_receipts == () + raw_inputs = { + item.column: item + for item in primary.inputs + if item.column in set(ACS_PUMS_EARNINGS_SOURCE_COLUMNS.values()) + } + assert set(raw_inputs) == set(ACS_PUMS_EARNINGS_SOURCE_COLUMNS.values()) + for declared in raw_inputs.values(): + assert declared.required_scope == "acs_source" + assert declared.producing_stage == US_LATE_EXTERNAL_STAGES[0] + assert declared.tolerated_absence_receipts == () + assert declared.alternatives[0][0].value_kind == "column_present" + + +def test_canonical_us_late_registry_declares_required_cross_producer_edges() -> None: + edges = set(CANONICAL_US_LATE_PRODUCER_SCHEDULE.edges) + + assert len(edges) == 71 + assert ( + US_LATE_ACS_EARNINGS_UNIVERSE_STAGE, + US_LATE_PRIMARY_PUF_STAGE, + ) in edges + assert ( + source_producer_name("with_us_pregnancy_inputs"), + source_producer_name("with_us_wic_claim_input"), + ) in edges + assert ( + source_producer_name("with_us_childcare_inputs"), + source_producer_name("with_us_adult_care_inputs"), + ) in edges + assert ( + transfer_producer_name("person", "puf_tax_itemization__batch_5"), + source_producer_name("with_us_adult_care_inputs"), + ) in edges + assert ( + transfer_producer_name("person", "puf_tax_itemization__batch_2"), + source_producer_name("with_us_education_inputs"), + ) in edges + assert { + consumer + for producer, consumer in edges + if producer == US_LATE_PRIMARY_PUF_STAGE and consumer.startswith("transfer:") + } == {group.name for group in CANONICAL_US_LATE_TRANSFER_GROUPS} + assert { + consumer + for producer, consumer in edges + if producer == US_LATE_PRIMARY_PUF_STAGE and consumer.startswith("source:") + } == { + source_producer_name(operator) + for operator in POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + } + assert CANONICAL_US_LATE_PRODUCER_SCHEDULE.waves[:2] == ( + (US_LATE_ACS_EARNINGS_UNIVERSE_STAGE,), + (US_LATE_PRIMARY_PUF_STAGE,), + ) + assert ( + US_LATE_SOURCE_FINALIZER_STAGE + in (CANONICAL_US_LATE_PRODUCER_SCHEDULE.waves[-1]) + ) + assert { + producer + for producer, consumer in edges + if consumer == US_LATE_SOURCE_FINALIZER_STAGE + } == { + source_producer_name(operator) + for operator in POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + } + assert any( + item.column == "@source_finalizer_execution_config" + and item.producing_stage == US_LATE_EXTERNAL_STAGES[0] + for item in CANONICAL_US_LATE_PRODUCER_REGISTRY[ + US_LATE_SOURCE_FINALIZER_STAGE + ].inputs + ) + assert { + (output.entity, output.column, output.coverage_scope) + for output in CANONICAL_US_LATE_PRODUCER_REGISTRY[ + US_LATE_SOURCE_FINALIZER_STAGE + ].outputs + } == { + ("person", column, "whole_pool") + for column in ("bank_account_assets", "bond_assets", "stock_assets") + } + + +def test_production_adult_care_contract_refuses_missing_sstb_before_callback() -> None: + contract = CANONICAL_US_LATE_PRODUCER_REGISTRY[ + source_producer_name("with_us_adult_care_inputs") + ] + sstb_input = next( + item + for item in contract.inputs + if item.column == "sstb_self_employment_income_before_lsr" + and item.producing_stage + == transfer_producer_name("person", "puf_tax_itemization__batch_5") + ) + invoked = False + + def callback() -> None: + nonlocal invoked + invoked = True + + with pytest.raises( + ValueError, + match=( + r"(?s)source:with_us_adult_care_inputs.*" + r"person\.sstb_self_employment_income_before_lsr.*43260 unfilled.*" + r"asec_source.*transfer:person/puf_tax_itemization__batch_5" + ), + ): + run_producer_when_ready( + contract, + callback, + unfilled_rows={ + item: 43_260 if item == sstb_input else 0 for item in contract.inputs + }, + invalid_rows={item: 0 for item in contract.inputs}, + absence_receipts={}, + ) + + assert invoked is False + + +def test_adult_care_transfer_declares_role_and_refuses_before_callback() -> None: + contract = CANONICAL_US_LATE_PRODUCER_REGISTRY[ + transfer_producer_name("person", "adult_care") + ] + role_input = next( + item + for item in contract.inputs + if item.column == "@effective:adult_care_tax_unit_role" + ) + assert role_input.alternatives == ( + (ProducerInputColumn("person", "tax_unit_role_input"),), + ) + assert role_input.required_scope == "whole_pool" + assert role_input.producing_stage == US_LATE_EXTERNAL_STAGES[0] + invoked = False + + def callback() -> None: + nonlocal invoked + invoked = True + + with pytest.raises( + ValueError, + match=( + r"(?s)transfer:person/adult_care.*" + r"person\.@effective:adult_care_tax_unit_role.*1 unfilled.*" + r"whole_pool.*post_clone_input_surface" + ), + ): + run_producer_when_ready( + contract, + callback, + unfilled_rows={ + item: 1 if item == role_input else 0 for item in contract.inputs + }, + invalid_rows={item: 0 for item in contract.inputs}, + absence_receipts={}, + ) + + assert invoked is False + + +def test_canonical_us_late_schedule_is_import_validated_and_byte_stable() -> None: + reverse_registry = OrderedDict( + reversed(tuple(CANONICAL_US_LATE_PRODUCER_REGISTRY.items())) + ) + reconstructed = derive_producer_schedule( + reverse_registry, + external_stages=US_LATE_EXTERNAL_STAGES, + ) + + assert reconstructed == CANONICAL_US_LATE_PRODUCER_SCHEDULE + receipt = us_late_producer_schedule_receipt() + assert receipt["schema_version"] == 16 + assert receipt["execution_receipt_contract"] == { + "version": 3, + "row_binding": ( + "declared_globally_reconciled_input_and_scope_exact_output_source_" + "and_primary_callback_resource_receipt_and_previous_execution_sha256" + ), + "virtual_resource_binding": ("exact_kind_specific_semantic_payload_and_sha256"), + "top_binding": ( + "entry_and_output_frame_sha256_execution_chain_source_" + "completion_and_nineteen_transfer_groups" + ), + "transition_authority": { + "authority_id": "us_stacked_late_producer_transition", + "metadata_key": "us_late_producer_transition_authority", + "version": 1, + "independent_digest_required": True, + }, + } + assert receipt["status"] == "derived_and_import_validated" + assert receipt["schedule_sha256"] == reconstructed.sha256 + assert receipt["producer_count"] == 38 + assert receipt["source_producer_count"] == 16 + assert receipt["transfer_group_count"] == 19 + assert receipt["transfer_target_count"] == 70 + assert receipt["order"][:2] == [ + US_LATE_ACS_EARNINGS_UNIVERSE_STAGE, + US_LATE_PRIMARY_PUF_STAGE, + ] + + +def test_every_post_clone_source_has_a_nonempty_full_input_inventory() -> None: + assert set(US_LATE_SOURCE_INPUT_INVENTORIES) == set( + POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + ) + for operator, inventory in US_LATE_SOURCE_INPUT_INVENTORIES.items(): + assert inventory.operator == operator + assert inventory.requirements + assert all(requirement.alternatives for requirement in inventory.requirements) + physical_columns = { + column.column + for requirement in inventory.requirements + for alternative in requirement.alternatives + for column in alternative + } + assert "@post_clone_source_execution_config" in physical_columns + assert "@weeks_unemployed_sidecar" not in physical_columns + assert "@education_assistance_sidecar" not in physical_columns + + +def test_acs_earnings_universe_declares_every_receipt_affecting_input() -> None: + inventory = US_LATE_ACS_EARNINGS_UNIVERSE_INPUT_INVENTORY + + assert len(inventory.requirements) == 10 + assert {requirement.label for requirement in inventory.requirements} == { + "age", + "support_channel", + "person_tax_unit_link", + "support_clone_index", + "stable_person_lineage", + "raw_source:WAGP", + "raw_source:SEMP", + "mapped_earnings:employment_income_before_lsr", + "mapped_earnings:self_employment_income_before_lsr", + "execution_config", + } + by_label = { + requirement.label: requirement for requirement in inventory.requirements + } + assert not by_label["raw_source:WAGP"].optional + assert not by_label["raw_source:SEMP"].optional + assert { + by_label[label].alternatives[0][0].value_kind + for label in ("raw_source:WAGP", "raw_source:SEMP") + } == {"column_present"} + assert by_label["mapped_earnings:employment_income_before_lsr"].optional + assert by_label["mapped_earnings:self_employment_income_before_lsr"].optional + lineage = next( + requirement + for requirement in inventory.requirements + if requirement.label == "stable_person_lineage" + ) + assert { + tuple((column.entity, column.column) for column in alternative) + for alternative in lineage.alternatives + } == { + (("person", "person_source_id"),), + (("person", "person_id"),), + } + + +def test_every_transfer_declares_predictors_and_optional_absence_receipts() -> None: + primary_inputs = { + item.column + for item in CANONICAL_US_LATE_PRODUCER_REGISTRY[ + US_LATE_PRIMARY_PUF_STAGE + ].inputs + } + assert { + "@effective:puf_donor", + "@effective:primary_qrf_bank", + "@effective:primary_puf_execution_config", + } <= primary_inputs + + for group in CANONICAL_US_LATE_TRANSFER_GROUPS: + contract = CANONICAL_US_LATE_PRODUCER_REGISTRY[group.name] + effective_inputs = { + item.column: item for item in contract.inputs if item.column.startswith("@") + } + assert { + "@effective:age", + "@effective:is_female", + "@effective:state_fips", + "@effective:resolved_person_weight", + "@effective:resolved_target_weight", + "@effective:optional_investment_income", + "@effective:late_transfer_model_config", + "@effective:late_transfer_target_bank", + } <= set(effective_inputs) + assert effective_inputs[ + "@effective:optional_investment_income" + ].tolerated_absence_receipts == ( + f"optional_input:{group.name}:optional_investment_income", + ) + + +def test_every_transfer_declares_complete_cross_grain_validation_surface() -> None: + entities = ( + "person", + "household", + "tax_unit", + "spm_unit", + "family", + "marital_unit", + ) + groups = entities[1:] + expected = { + *((entity, f"{entity}_support_channel") for entity in entities), + *((entity, f"{entity}_support_clone_index") for entity in entities), + *(("person", f"person_{entity}_id") for entity in groups), + *((entity, f"{entity}_id") for entity in groups), + ("person", "person_id"), + ("person", "person_spine_source_id"), + ("person", "person_source_id"), + ("household", "household_spine_source_id"), + ("household", "household_source_id"), + ("household", "TYPEHUGQ"), + ("household", "@resolved_weight"), + ("frame", "@us_spine_assembly_manifest"), + ("frame", "@us_stacked_spine_manifest"), + ("frame", "@us_puf_clone_attachment_manifest"), + } + for group in CANONICAL_US_LATE_TRANSFER_GROUPS: + inventory = US_LATE_TRANSFER_INPUT_INVENTORIES[group.name] + physical = { + (column.entity, column.column) + for requirement in inventory.requirements + for alternative in requirement.alternatives + for column in alternative + } + assert expected <= physical + + clone_columns = { + (column.entity, column.column, column.value_kind) + for requirement in inventory.requirements + for alternative in requirement.alternatives + for column in alternative + if column.column.endswith("_support_clone_index") + } + assert clone_columns == { + (entity, f"{entity}_support_clone_index", "finite_numeric") + for entity in entities + } + + +def test_every_origin_exclusive_raw_input_has_its_native_scope() -> None: + acs_raw_inputs = { + *(("household", column) for column in acs_pums_module._HOUSEHOLD_FRAME_COLUMNS), + *(("person", column) for column in acs_pums_module._PERSON_REQUIRED), + } + asec_person_raw_columns = { + "A_HSCOL", + "A_MJOCC", + "CAID", + "CHAMPVA", + "CHSP_VAL", + "CSP_VAL", + "DIS_SC1", + "DIS_SC2", + "DIS_VAL1", + "DIS_VAL2", + "DST_SC1", + "DST_SC2", + "DST_SC3", + "DST_SC4", + "DST_VAL1", + "DST_VAL2", + "DST_VAL3", + "DST_VAL4", + "ED_VAL", + "IHSFLG", + "I_ERNVAL", + "I_SEVAL", + "LKWEEKS", + "MCARE", + "MIL", + "PEAFEVER", + "PEDISDRS", + "PEINUSYR", + "PEIO1COW", + "PENATVTY", + "PEN_SC1", + "PEN_SC2", + "PERIDNUM", + "PRCITSHP", + "RESNSS1", + "RESNSS2", + "RETCB_VAL", + "SPM_CAPHOUSESUB", + "SPM_CHILDCAREXPNS", + "SPM_ENGVAL", + "SSI_YN", + "SS_YN", + "UC_VAL", + "WC_VAL", + } + required_scope = { + **{key: "acs_source" for key in acs_raw_inputs}, + **{("person", column): "asec_source" for column in asec_person_raw_columns}, + ("household", "H_TENURE"): "asec_source", + } + expected_counts = { + ("household", "TYPEHUGQ"): 39, + ("person", "MCARE"): 2, + ("person", "PERIDNUM"): 18, + ("person", "SEMP"): 2, + ("person", "WAGP"): 2, + **{ + ("person", column): 1 + for column in asec_person_raw_columns + if column + not in { + "DST_SC3", + "DST_SC4", + "DST_VAL3", + "DST_VAL4", + "MCARE", + "PERIDNUM", + } + }, + } + + observed: Counter[tuple[str, str, str]] = Counter() + receipts: dict[tuple[str, str], set[str]] = {} + for contract in CANONICAL_US_LATE_PRODUCER_REGISTRY.values(): + for requirement in contract.inputs: + for alternative in requirement.alternatives: + for column in alternative: + key = (column.entity, column.column) + if key not in required_scope: + continue + observed[ + (column.entity, column.column, requirement.required_scope) + ] += 1 + receipts.setdefault(key, set()).update( + requirement.tolerated_absence_receipts + ) + + assert observed == Counter( + {(*key, required_scope[key]): count for key, count in expected_counts.items()} + ) + assert sum(observed.values()) == 101 + assert sum(count for key, count in observed.items() if key[2] == "acs_source") == 43 + assert ( + sum(count for key, count in observed.items() if key[2] == "asec_source") == 58 + ) + assert receipts[("household", "TYPEHUGQ")] == set() + + execution_identity = acs_transfer_module.acs_transfer_execution_contract_identity() + assert execution_identity["schema_version"] == 2 + assert execution_identity["housing"]["head_source_precedence"] == [ + {"source": "is_household_head", "head_codes": [True]}, + {"source": "A_EXPRRP", "head_codes": [1, 2]}, + {"source": "A_LINENO", "head_codes": [1]}, + ] + assert execution_identity["housing"]["tenure_source_precedence"] == [ + "tenure_type", + "spm_unit_tenure_type", + ] + + +def test_production_registry_preserves_finite_numeric_input_kinds() -> None: + cases = ( + ( + US_LATE_PRIMARY_PUF_STAGE, + "@effective:employment_income", + "person", + "employment_income_before_lsr", + ), + ( + source_producer_name("with_us_adult_care_inputs"), + "@effective:sstb_earned_income", + "person", + "sstb_self_employment_income_before_lsr", + ), + ( + source_producer_name("with_us_education_inputs"), + "@effective:qualified_tuition", + "person", + "qualified_tuition_expenses", + ), + ( + transfer_producer_name("person", "adult_care"), + "@effective:optional_employment_income", + "person", + "employment_income_before_lsr", + ), + ) + + for producer, input_name, entity, column in cases: + effective_input = next( + item + for item in CANONICAL_US_LATE_PRODUCER_REGISTRY[producer].inputs + if item.column == input_name + ) + declared_column = next( + item + for alternative in effective_input.alternatives + for item in alternative + if (item.entity, item.column) == (entity, column) + ) + assert declared_column.value_kind == "finite_numeric" + + +def test_source_contracts_match_strict_runtime_input_semantics() -> None: + adult = CANONICAL_US_LATE_PRODUCER_REGISTRY[ + source_producer_name("with_us_adult_care_inputs") + ] + adult_inputs = {item.column: item for item in adult.inputs} + assert adult_inputs["@effective:support_role"].alternatives == ( + ( + next( + column + for alternative in adult_inputs["@effective:support_role"].alternatives + for column in alternative + if column.column == "person_support_channel" + ), + next( + column + for alternative in adult_inputs["@effective:support_role"].alternatives + for column in alternative + if column.column == "person_support_clone_index" + ), + ), + ) + for logical_input, physical_column in ( + ("@effective:raw_person:PEDISDRS", "PEDISDRS"), + ( + "@effective:raw_person:is_full_time_college_student", + "is_full_time_college_student", + ), + ): + requirement = adult_inputs[logical_input] + assert { + column.value_kind + for alternative in requirement.alternatives + for column in alternative + if column.column == physical_column + } == {"finite_numeric"} + + education = CANONICAL_US_LATE_PRODUCER_REGISTRY[ + source_producer_name("with_us_education_inputs") + ] + education_source = next( + item + for item in education.inputs + if item.column == "@effective:education_source" + ) + ed_val = next( + column + for alternative in education_source.alternatives + for column in alternative + if column.column == "ED_VAL" + ) + assert ed_val.value_kind == "finite_numeric" + + +def test_transfer_numeric_predictor_alternatives_are_all_finite() -> None: + numeric_requirements = { + "@effective:optional_social_security_income", + "@effective:optional_retirement_income", + "@effective:optional_investment_income", + } + for group in CANONICAL_US_LATE_TRANSFER_GROUPS: + contract = CANONICAL_US_LATE_PRODUCER_REGISTRY[group.name] + inputs = {item.column: item for item in contract.inputs} + for logical_input in numeric_requirements: + assert { + column.value_kind + for alternative in inputs[logical_input].alternatives + for column in alternative + } == {"finite_numeric"} + + +def test_source_numeric_input_audit_is_fully_executable() -> None: + expected_finite = { + "with_us_prior_year_income_inputs": { + "source_year", + "WSAL_VAL", + "SEMP_VAL", + "I_ERNVAL", + "I_SEVAL", + "employment_income_last_year", + "self_employment_income_last_year", + }, + "with_us_medicare_take_up_input": {"MCARE"}, + "with_us_pregnancy_inputs": {"A_SEX", "A_AGE"}, + "with_us_wic_claim_input": { + "age", + "is_female", + "is_pregnant", + "own_children_in_household", + "person_family_id", + }, + "impute_us_housing_assistance_to_puf_support": { + "receives_housing_assistance", + "takes_up_housing_assistance_if_eligible", + }, + "with_us_child_support_inputs": {"CSP_VAL", "CHSP_VAL"}, + "with_us_disability_benefits": { + "DIS_VAL1", + "DIS_SC1", + "DIS_VAL2", + "DIS_SC2", + }, + "with_us_workers_compensation": {"WC_VAL"}, + "with_us_weeks_unemployed": { + "source_year", + "PERIDNUM", + "LKWEEKS", + "age", + "A_AGE", + "is_male", + "is_female", + "A_SEX", + "tax_unit_is_joint", + "is_tax_unit_head", + "is_tax_unit_spouse", + "is_tax_unit_dependent", + "unemployment_compensation", + "UC_VAL", + }, + "with_us_childcare_inputs": {"person_spm_unit_id", "SPM_CHILDCAREXPNS"}, + "with_us_adult_care_inputs": { + "PEDISDRS", + "is_full_time_college_student", + "person_id", + "person_support_clone_index", + }, + "with_us_energy_subsidy_input": {"person_spm_unit_id", "SPM_ENGVAL"}, + "with_us_retirement_contribution_inputs": { + "RETCB_VAL", + "WSAL_VAL", + "SEMP_VAL", + }, + "with_us_retirement_distribution_inputs": { + "DST_SC1", + "DST_VAL1", + "DST_SC2", + "DST_VAL2", + "DST_SC1_YNG", + "DST_VAL1_YNG", + "DST_SC2_YNG", + "DST_VAL2_YNG", + "taxable_ira_distributions", + }, + "with_us_immigration_inputs": { + "PRCITSHP", + "PEINUSYR", + "PENATVTY", + "A_AGE", + "A_MARITL", + "A_SPOUSE", + "A_HSCOL", + "WSAL_VAL", + "SEMP_VAL", + "MCARE", + "CAID", + "IHSFLG", + "CHAMPVA", + "MIL", + "PEN_SC1", + "PEN_SC2", + "RESNSS1", + "RESNSS2", + "SS_YN", + "SSI_YN", + "PEIO1COW", + "A_MJOCC", + "PEAFEVER", + "SPM_CAPHOUSESUB", + }, + "with_us_education_inputs": {"ED_VAL", "qualified_tuition_expenses"}, + } + assert set(expected_finite) == set(US_LATE_SOURCE_INPUT_INVENTORIES) + for operator, expected_columns in expected_finite.items(): + inventory = US_LATE_SOURCE_INPUT_INVENTORIES[operator] + finite_columns = { + column.column + for requirement in inventory.requirements + for alternative in requirement.alternatives + for column in alternative + if column.value_kind == "finite_numeric" + } + assert expected_columns <= finite_columns, operator + + common_role_operators = { + "with_us_prior_year_income_inputs", + "impute_us_housing_assistance_to_puf_support", + "with_us_child_support_inputs", + "with_us_disability_benefits", + "with_us_workers_compensation", + "with_us_childcare_inputs", + "with_us_energy_subsidy_input", + "with_us_retirement_contribution_inputs", + "with_us_retirement_distribution_inputs", + } + for operator in common_role_operators: + sex = next( + requirement + for requirement in US_LATE_SOURCE_INPUT_INVENTORIES[operator].requirements + if requirement.label == "sex" + ) + assert { + column.value_kind + for alternative in sex.alternatives + for column in alternative + } == {"finite_numeric"} + + +def test_late_target_dependency_kinds_partition_51_numeric_17_boolean_2_string() -> ( + None +): + string_targets = {"ssn_card_type", "immigration_status_str"} + boolean_targets = { + "is_incapable_of_self_care", + "is_pregnant", + "estate_income_would_be_qualified", + "farm_operations_income_would_be_qualified", + "farm_rent_income_would_be_qualified", + "partnership_s_corp_income_would_be_qualified", + "rental_income_would_be_qualified", + "self_employment_income_would_be_qualified", + "sstb_self_employment_income_would_be_qualified", + "business_is_sstb", + "attends_eligible_educational_institution_for_american_opportunity_credit", + "has_american_opportunity_credit_1098_t_or_exception", + "has_american_opportunity_credit_institution_ein", + "is_enrolled_at_least_half_time_for_american_opportunity_credit", + "is_pursuing_credential_for_american_opportunity_credit", + "takes_up_medicare_if_eligible", + "would_claim_wic", + } + observed: dict[str, set[str]] = {} + for group in CANONICAL_US_LATE_TRANSFER_GROUPS: + contract = CANONICAL_US_LATE_PRODUCER_REGISTRY[group.name] + for target in group.targets: + direct_inputs = [item for item in contract.inputs if item.column == target] + assert direct_inputs + observed[target] = { + column.value_kind + for item in direct_inputs + for alternative in item.alternatives + for column in alternative + } + numeric_targets = set(observed) - boolean_targets - string_targets + assert (len(numeric_targets), len(boolean_targets), len(string_targets)) == ( + 51, + 17, + 2, + ) + assert all(observed[target] == {"finite_numeric"} for target in numeric_targets) + assert all( + observed[target] == {"non_null"} for target in boolean_targets | string_targets + ) diff --git a/packages/microcosm-build/tests/test_us_multispine_pool.py b/packages/microcosm-build/tests/test_us_multispine_pool.py index 3b831342..35e1b826 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool.py @@ -21,6 +21,10 @@ from microcosm.build.us_runtime import multispine_pool as multispine_pool_module from microcosm.build.us_runtime import prior_year_income as prior_year_income_module from microcosm.build.us_runtime import puf_support as puf_support_module +from microcosm.build.us_runtime import stacked_spine as stacked_spine_module +from microcosm.build.us_runtime.acs_income_universe import ( + ACS_PUMS_EARNINGS_UNIVERSE_PERSON_INPUTS, +) from microcosm.build.us_runtime.acs_transfer import ( declared_acs_transfer_target_families, ) @@ -28,29 +32,39 @@ POOL_CHECKPOINT_STAGE_ORDER, POOL_DEFERRED_TRANSFER_INPUTS, POOL_DERIVE_OPERATOR_ORDER, + POOL_ENGINE_INPUT_PROJECTION_CONTRACT, POOL_OPERATOR_CONTRACTS, POOL_OPERATOR_ORDER, POOL_POST_CLONE_SOURCE_OPERATOR_ORDER, POOL_PRE_CLONE_SOURCE_OPERATOR_ORDER, + POOL_REMAINING_STAGE_INPUT_MANIFEST_SHA256, POOL_SOURCE_OPERATOR_CONTRACTS, POOL_SOURCE_OPERATOR_ORDER, POOL_SPINE_AGREEMENT_REGISTRY, + POOL_SSI_DEPENDENCY_CONTRACT, MultispinePoolCheckpoint, MultispinePoolResult, PoolInputSurfaceEntry, + PoolRemainingStageInput, PoolStageOutput, _complete_schedule_d_input, + finalize_multispine_source_inputs, materialize_multispine_agreement_outputs, materialize_pool_deferred_transfer_inputs, + pool_engine_input_projection_receipt, pool_input_surface, pool_post_puf_puf_producer_target_families, pool_post_puf_source_producer_target_families, pool_post_puf_transfer_target_families, pool_pre_clone_gap_fill_target_families, + pool_remaining_stage_input_manifest, + pool_remaining_stage_input_manifest_receipt, + pool_ssi_dependency_closure, pool_transfer_target_families, prepare_multispine_puf_predictors, prepare_multispine_source_inputs_for_clone, run_multispine_pool_path, + run_multispine_post_clone_source_operator, seed_multispine_pool_inputs, ) from microcosm.build.us_runtime.operator_boundary import ( @@ -84,6 +98,7 @@ from microcosm.build.us_runtime.take_up_contract import load_take_up_contract from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights from microcosm.frame.adapters.policyengine_us import ( + PolicyEngineUSEngine, PolicyEngineUSVariableMetadataIndex, ) @@ -149,6 +164,170 @@ def _installed_variable_metadata_index() -> PolicyEngineUSVariableMetadataIndex: pytest.skip("requires the policyengine-us [us] extra") +def _overlap_person_table() -> pd.DataFrame: + return pd.DataFrame( + { + "person_id": np.asarray([1, 2, 3, 4, 5], dtype=np.int64), + "person_source_id": np.asarray([10, 10, 10, 20, 20], dtype=np.int64), + "person_support_channel": ["asec"] * 5, + "person_support_clone_index": np.asarray([0, 1, 2, 0, 1], dtype=np.int64), + "qualified_tuition_expenses": np.asarray( + [10.0, 20.0, 20.0, 30.0, 40.0], dtype=np.float64 + ), + "traditional_ira_contributions_desired": np.asarray( + [1.0, 101.25, 999.0, 2.0, 202.5], dtype=np.float64 + ), + "self_employed_pension_contributions_desired": np.asarray( + [3.0, -0.0, 777.0, 4.0, 404.5], dtype=np.float64 + ), + } + ) + + +def test_source_overlap_finalizer_mirrors_retirement_tail_bytes() -> None: + before = _overlap_person_table() + after = before.copy(deep=True) + + finalized, receipt = multispine_pool_module._finalize_source_overlap_output( + before, + after, + operator_name="with_us_retirement_contribution_inputs", + ) + + clone_index = finalized["person_support_clone_index"] + for target in ( + "traditional_ira_contributions_desired", + "self_employed_pension_contributions_desired", + ): + parent = finalized.loc[clone_index.eq(1)].set_index("person_source_id")[target] + tail = finalized.loc[clone_index.eq(2)].set_index("person_source_id")[target] + expected = parent.loc[tail.index].to_numpy() + actual = tail.to_numpy() + assert actual.dtype == expected.dtype + assert actual.tobytes() == expected.tobytes() + assert receipt["targets"][f"person.{target}"]["action"] == ( + "byte_exact_clone_1_mirror" + ) + assert receipt["targets"][f"person.{target}"]["mirrored_clone_2_rows"] == 1 + assert finalized.loc[clone_index.eq(1)].equals(after.loc[clone_index.eq(1)]) + assert receipt["passed"] is True + + +@pytest.mark.parametrize("mutation", ["value", "dtype"]) +def test_source_overlap_finalizer_rejects_education_tuition_write( + mutation: str, +) -> None: + before = _overlap_person_table() + after = before.copy(deep=True) + if mutation == "value": + after.loc[0, "qualified_tuition_expenses"] += 1.0 + else: + after["qualified_tuition_expenses"] = after[ + "qualified_tuition_expenses" + ].astype(np.float32) + + with pytest.raises(ValueError, match="qualified_tuition_expenses.*byte identity"): + multispine_pool_module._finalize_source_overlap_output( + before, + after, + operator_name="with_us_education_inputs", + ) + + +def test_postclone_chain_receipts_education_tuition_passthrough_noop() -> None: + asec = _source_frame() + asec_tables = {entity: asec.table(entity).copy() for entity in asec.entities} + asec_tables["person"]["PERIDNUM"] = ["asec-1", "asec-2"] + asec_tables["person"]["qualified_tuition_expenses"] = np.asarray( + [125.5, -0.0], dtype=np.float64 + ) + asec = Frame( + asec_tables, + asec.schema, + {"household": asec.weights_for("household")}, + asec.strata, + ) + acs = _source_frame(offset=100.0) + assembled = assemble_spines( + {"asec": asec, "acs": acs}, + household_mass_shares={"asec": 0.5, "acs": 0.5}, + ) + cloned = clone_us_frame_for_puf_support(assembled) + + def education(available: Frame) -> Frame: + person = available.table("person").copy() + person["educational_assistance"] = 0.0 + return _replace_person(available, person) + + completed = multispine_pool_module._run_source_operator_chain( + cloned, + phase="post_clone", + operator_names=("with_us_education_inputs",), + operators={"with_us_education_inputs": education}, + output_families={ + "education_inputs": { + "person": frozenset({"educational_assistance"}), + } + }, + ) + + source = cloned.table("person")["qualified_tuition_expenses"] + observed = completed.frame.table("person")["qualified_tuition_expenses"] + assert observed.dtype == source.dtype + assert observed.to_numpy().tobytes() == source.to_numpy().tobytes() + suboperator = completed.receipt["suboperators"][0] + assert suboperator["output_columns"] == {"person": ["educational_assistance"]} + overlap = suboperator["overlap_ownership"] + assert overlap["targets"]["person.qualified_tuition_expenses"]["action"] == ( + "consume_only_byte_exact_noop" + ) + + +def test_postclone_chain_rejects_education_omitting_tuition_passthrough() -> None: + asec = _source_frame() + asec_tables = {entity: asec.table(entity).copy() for entity in asec.entities} + asec_tables["person"]["PERIDNUM"] = ["asec-1", "asec-2"] + asec_tables["person"]["qualified_tuition_expenses"] = np.asarray( + [125.5, -0.0], dtype=np.float64 + ) + asec = Frame( + asec_tables, + asec.schema, + {"household": asec.weights_for("household")}, + asec.strata, + ) + acs = _source_frame(offset=100.0) + cloned = clone_us_frame_for_puf_support( + assemble_spines( + {"asec": asec, "acs": acs}, + household_mass_shares={"asec": 0.5, "acs": 0.5}, + ) + ) + + def education_omitting_passthrough(available: Frame) -> Frame: + person = available.table("person").drop(columns="qualified_tuition_expenses") + person["educational_assistance"] = 0.0 + return _replace_person(available, person) + + with pytest.raises( + ValueError, + match="education overlap target.*qualified_tuition_expenses.*absent", + ): + multispine_pool_module._run_source_operator_chain( + cloned, + phase="post_clone", + operator_names=("with_us_education_inputs",), + operators={ + "with_us_education_inputs": education_omitting_passthrough, + }, + output_families={ + "education_inputs": { + "person": frozenset({"educational_assistance"}), + } + }, + ) + + _EXPECTED_SOURCE_OPERATOR_WRAPPERS = { "with_us_hours_worked_inputs": "_with_gated_us_hours_worked_inputs", "with_us_qbi_input_reconciliation": "reconcile_qbi_with_receipt", @@ -1015,8 +1194,8 @@ def keys(target_families): assert early.isdisjoint(late) assert early | late == full assert len(puf_producers) == 43 - assert len(source_producers) == 30 - assert len(puf_producers & source_producers) == 3 + assert len(source_producers) == 29 + assert len(puf_producers & source_producers) == 2 assert puf_producers | source_producers == late assert ("person", "source_operator_cps_carried", "strike_benefits") in early assert ("person", "model_required_boolean", "is_pregnant") in late @@ -1035,6 +1214,11 @@ def keys(target_families): "model_required_boolean", "is_pregnant", ) in source_producers + assert ( + "person", + "puf_tax_itemization", + "qualified_tuition_expenses", + ) not in source_producers def test_pool_input_surface_normalizes_all_four_source_registries() -> None: @@ -1153,6 +1337,359 @@ def test_pool_input_surface_rejects_primary_qrf_target_without_entity( pool_input_surface() +def test_ssi_static_dependency_closure_matches_pinned_engine_graph() -> None: + closure = pool_ssi_dependency_closure(_installed_variable_metadata_index()) + + assert closure.engine_version == POOL_SSI_DEPENDENCY_CONTRACT.engine_version + assert closure.root == "ssi" + assert len(closure.input_leaves) == 55 + assert len(closure.formula_nodes) == 62 + assert len(closure.edges) == 186 + assert closure.sha256 == POOL_SSI_DEPENDENCY_CONTRACT.sha256 + assert closure.input_leaves == ( + "age", + "alimony_income", + "bank_account_assets", + "bond_assets", + "child_support_received", + "disability_benefits", + "employment_income_before_lsr", + "financial_assistance", + "gi_cash_assistance", + "immigration_status_str", + "is_blind", + "is_disabled", + "is_full_time_college_student", + "is_separated", + "keogh_distributions", + "meets_ssi_disability_criteria", + "non_qualified_dividend_income", + "own_children_in_household", + "qualified_dividend_income", + "rental_income", + "self_employment_income_before_lsr", + "social_security_dependents", + "social_security_disability", + "social_security_retirement", + "social_security_survivors", + "ssi_lives_in_another_persons_household", + "ssi_lives_in_medical_treatment_facility", + "ssi_medicaid_pays_majority_of_care", + "ssi_others_pay_all_meals", + "ssi_qualifying_quarters_earnings", + "ssi_receives_food_from_others", + "ssi_receives_outside_shelter_support", + "ssi_receives_shelter_from_others_in_household", + "ssi_shelter_support_value", + "sstb_self_employment_income_before_lsr", + "stock_assets", + "survivor_benefits", + "takes_up_ssi_if_eligible", + "tax_exempt_401k_distributions", + "tax_exempt_403b_distributions", + "tax_exempt_interest_income", + "tax_exempt_ira_distributions", + "tax_exempt_private_pension_income", + "tax_exempt_public_pension_income", + "tax_exempt_sep_distributions", + "taxable_401k_distributions", + "taxable_403b_distributions", + "taxable_interest_income", + "taxable_ira_distributions", + "taxable_private_pension_income", + "taxable_public_pension_income", + "taxable_sep_distributions", + "unemployment_compensation", + "veterans_benefits", + "workers_compensation", + ) + + +def test_remaining_stage_manifest_covers_every_derive_read() -> None: + manifest = pool_remaining_stage_input_manifest(_installed_variable_metadata_index()) + derive = [entry for entry in manifest if entry.stage == "derive"] + by_consumer = { + consumer: { + (entry.entity, entry.variable) + for entry in derive + if entry.consumer == consumer + } + for consumer in {entry.consumer for entry in derive} + } + + assert by_consumer == { + "prepare_stacked_tail_derivation": { + ("person", "person_support_clone_index"), + ("person", "schedule_d_capital_gain_distributions"), + }, + "_complete_schedule_d_input": { + ("person", "long_term_capital_gains_before_response"), + ("person", "non_sch_d_capital_gains"), + ("person", "person_tax_unit_id"), + ("person", "schedule_d_capital_gain_distributions"), + ("tax_unit", "tax_unit_id"), + }, + "with_us_qbi_input_reconciliation": { + *{("person", variable) for variable in US_QBI_OUTPUT_COLUMNS}, + ("person", "self_employment_income_before_lsr"), + ("person", "partnership_income"), + ("person", "s_corp_income"), + ("person", "estate_income"), + ("person", "non_qualified_dividend_income"), + ("person", "age"), + ("person", "SEMP"), + ("person", "person_tax_unit_id"), + ("person", "person_support_clone_index"), + ("person", "person_support_channel"), + ("person", "person_source_id"), + ("person", "person_id"), + }, + } + s_corp = next( + entry + for entry in derive + if entry.consumer == "with_us_qbi_input_reconciliation" + and entry.variable == "s_corp_income" + ) + assert s_corp == PoolRemainingStageInput( + stage="derive", + consumer="with_us_qbi_input_reconciliation", + entity="person", + variable="s_corp_income", + execution_scope="whole_pool", + provision="primary_puf_exact_zero_universe", + available_by="transferred", + ) + assert set(ACS_PUMS_EARNINGS_UNIVERSE_PERSON_INPUTS).issubset( + { + variable + for entity, variable in by_consumer["with_us_qbi_input_reconciliation"] + if entity == "person" + } + ) + + +def test_remaining_stage_manifest_covers_seed_programs_and_structure() -> None: + manifest = pool_remaining_stage_input_manifest(_installed_variable_metadata_index()) + seed = [entry for entry in manifest if entry.stage == "seed"] + program_names = {program.variable for program in load_take_up_contract().programs} + programs = [entry for entry in seed if entry.variable in program_names] + + assert len(programs) == len(program_names) == 13 + assert {entry.variable for entry in programs} == program_names + assert Counter(entry.provision for entry in programs) == Counter( + { + "administrative_seed_or_preserved_input": 2, + "transferred_or_preserved_input": 2, + "preserved_input_or_disclosed_engine_default": 9, + } + ) + structural = { + (entry.entity, entry.variable, entry.provision) + for entry in seed + if entry.variable not in program_names + } + assert structural == { + *{ + ( + entity, + f"{entity}_source_id", + "assembly_support_source_identity", + ) + for entity in ( + "person", + "household", + "tax_unit", + "spm_unit", + "family", + "marital_unit", + ) + }, + ("person", "age", "assembled_native_person_input"), + ("person", "person_spm_unit_id", "frame_membership"), + ("person", "person_tax_unit_id", "frame_membership"), + ("person", "source_household_id", "optional_assembled_source_identity"), + ("person", "source_person_id", "optional_assembled_source_identity"), + ("person", "source_year", "optional_assembled_source_identity"), + ( + "spm_unit", + "", + "frame_resolve_weights_from_household_weight", + ), + ("spm_unit", "spm_unit_id", "frame_entity_id"), + ( + "tax_unit", + "", + "frame_resolve_weights_from_household_weight", + ), + ("tax_unit", "tax_unit_id", "frame_entity_id"), + } + + +def test_remaining_stage_manifest_provisions_every_ssi_leaf_by_seed() -> None: + index = _installed_variable_metadata_index() + manifest = pool_remaining_stage_input_manifest(index) + closure = pool_ssi_dependency_closure(index) + leaves = [ + entry for entry in manifest if entry.consumer == "ssi_static_dependency_closure" + ] + + assert len(leaves) == 55 + assert tuple(sorted(entry.variable for entry in leaves)) == closure.input_leaves + assert Counter(entry.provision for entry in leaves) == Counter( + { + "assembled_native_person_input": 1, + "materialized_pool_input_surface": 32, + "seed_stage_program_contract": 1, + "declared_deferred_null_input": 3, + "declared_absent_engine_input": 18, + } + ) + transferred_complete = sum( + entry.provision + in {"assembled_native_person_input", "materialized_pool_input_surface"} + for entry in leaves + ) + deferred = sum( + entry.provision == "declared_deferred_null_input" for entry in leaves + ) + transferred_absent = len(leaves) - transferred_complete - deferred + seeded_complete = transferred_complete + sum( + entry.provision == "seed_stage_program_contract" for entry in leaves + ) + seeded_absent = len(leaves) - seeded_complete - deferred + assert (transferred_complete, deferred, transferred_absent) == (33, 3, 19) + assert (seeded_complete, deferred, seeded_absent) == (34, 3, 18) + assert all( + entry.fallback is not None + for entry in leaves + if entry.provision + in { + "seed_stage_program_contract", + "declared_deferred_null_input", + "declared_absent_engine_input", + } + ) + + +def test_remaining_stage_manifest_enumerates_every_simulation_projection_input() -> ( + None +): + index = _installed_variable_metadata_index() + manifest = pool_remaining_stage_input_manifest(index) + projection = [ + entry for entry in manifest if entry.consumer == "_simulation_projection" + ] + + assert len(projection) == POOL_ENGINE_INPUT_PROJECTION_CONTRACT.input_count == 863 + assert {(entry.entity, entry.variable) for entry in projection} == { + (index.variable_metadata(variable).entity, variable) + for variable in index.variables() + } + assert Counter(entry.provision for entry in projection) == Counter( + { + "materialized_pool_input_surface": 123, + "seed_stage_program_contract": 13, + "declared_deferred_null_input": 3, + "assembled_native_engine_input": 5, + "frame_structural_engine_input": 10, + "preserved_stacked_engine_input": 4, + "derived_schedule_d_input": 1, + "declared_absent_engine_input": 704, + } + ) + preserved = { + (entry.entity, entry.variable, entry.fallback) + for entry in projection + if entry.provision == "preserved_stacked_engine_input" + } + assert preserved == { + ( + "person", + "is_related_to_head_or_spouse", + "ephemeral_simulation_projection_engine_default_if_present_null", + ), + ( + "household", + "puma", + "ephemeral_simulation_projection_engine_default_for_null", + ), + ( + "person", + "ssi_reported", + "ephemeral_simulation_projection_engine_default_for_null", + ), + ( + "household", + "state_fips", + "ephemeral_simulation_projection_engine_default_if_present_null", + ), + } + assert all(entry.fallback is not None for entry in projection) + + +def test_simulation_projection_defaults_match_pinned_engine_surface() -> None: + pytest.importorskip("policyengine_us") + receipt = pool_engine_input_projection_receipt(PolicyEngineUSEngine()) + + assert receipt == { + "engine_version": "1.764.6", + "input_count": 863, + "default_count": 863, + "defaults_sha256": (POOL_ENGINE_INPUT_PROJECTION_CONTRACT.defaults_sha256), + } + + +def test_remaining_stage_manifest_is_unique_complete_and_stable() -> None: + manifest = pool_remaining_stage_input_manifest(_installed_variable_metadata_index()) + + assert len(manifest) == 993 + assert Counter(entry.stage for entry in manifest) == Counter( + {"derive": 34, "seed": 29, "simulate": 930} + ) + assert len( + { + (entry.stage, entry.consumer, entry.entity, entry.variable) + for entry in manifest + } + ) == len(manifest) + assert all(entry.provision and entry.available_by for entry in manifest) + + receipt = pool_remaining_stage_input_manifest_receipt( + _installed_variable_metadata_index() + ) + assert receipt["entry_count"] == 993 + assert receipt["stage_counts"] == { + "derive": 34, + "seed": 29, + "simulate": 930, + } + assert receipt["engine_input_projection_contract"] == { + "engine_version": "1.764.6", + "input_count": 863, + "default_count": 863, + "sha256": POOL_ENGINE_INPUT_PROJECTION_CONTRACT.sha256, + "defaults_sha256": POOL_ENGINE_INPUT_PROJECTION_CONTRACT.defaults_sha256, + } + assert receipt["manifest_sha256"] == POOL_REMAINING_STAGE_INPUT_MANIFEST_SHA256 + assert len(receipt["sha256"]) == 64 + + +def test_remaining_stage_manifest_rejects_unreviewed_content_drift( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr( + multispine_pool_module, + "POOL_REMAINING_STAGE_INPUT_MANIFEST_SHA256", + "0" * 64, + ) + + with pytest.raises(ValueError, match="Remaining-stage input manifest drifted"): + pool_remaining_stage_input_manifest_receipt( + _installed_variable_metadata_index() + ) + + class _ProducerDtypeFittedQRF: def __init__( self, @@ -1649,7 +2186,7 @@ def test_every_pool_transfer_family_accepts_its_produced_physical_dtype( assert sum(calls[name] for name in POOL_SOURCE_OPERATOR_ORDER) == 22 -def test_object_backed_is_female_reaches_production_transfer_fit( +def test_object_backed_is_female_becomes_nullable_before_production_transfer_fit( monkeypatch: pytest.MonkeyPatch, ) -> None: calls: Counter[str] = Counter() @@ -1668,7 +2205,7 @@ def test_object_backed_is_female_reaches_production_transfer_fit( assert all(isinstance(value, (bool, np.bool_)) for value in assembled.dropna()) for stage in (stages["prepared"], produced): is_female = stage.person["is_female"] - assert pd.api.types.is_object_dtype(is_female.dtype) + assert is_female.dtype == pd.BooleanDtype() assert not is_female.isna().any() assert all(isinstance(value, (bool, np.bool_)) for value in is_female) @@ -1678,7 +2215,7 @@ def test_object_backed_is_female_reaches_production_transfer_fit( ) assert role == "puf_tax_detail" assert len(donor.person) == 6 - assert pd.api.types.is_object_dtype(donor.person["is_female"].dtype) + assert donor.person["is_female"].dtype == pd.BooleanDtype() monkeypatch.setattr( acs_transfer_module, @@ -1932,9 +2469,393 @@ def test_every_source_operator_has_an_executable_clone_phase_contract() -> None: } == set(POOL_DERIVE_OPERATOR_ORDER) +def test_single_post_clone_source_entrypoint_dispatches_one_operator( + monkeypatch: pytest.MonkeyPatch, +) -> None: + observed: list[tuple[str, tuple[str, ...], tuple[str, ...]]] = [] + + def observe_guarded_chain( + frame: Frame, + *, + phase: str, + operator_names: tuple[str, ...], + operators: dict[str, Callable[[Frame], Frame]], + **_kwargs: object, + ) -> PoolStageOutput: + observed.append((phase, operator_names, tuple(operators))) + return PoolStageOutput( + frame, + { + "phase": phase, + "operator_order": list(operator_names), + "suboperators": [{"operator": operator_names[0], "order_index": 0}], + }, + ) + + monkeypatch.setattr( + multispine_pool_module, + "_run_source_operator_chain", + observe_guarded_chain, + ) + + result = run_multispine_post_clone_source_operator( + _source_frame(), + "with_us_adult_care_inputs", + ) + + assert result.receipt["operator_order"] == ["with_us_adult_care_inputs"] + assert observed == [ + ( + "post_clone", + ("with_us_adult_care_inputs",), + ("with_us_adult_care_inputs",), + ) + ] + + +def test_single_post_clone_source_entrypoint_rejects_unknown_operator_before_run( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr( + multispine_pool_module, + "_run_source_operator_chain", + lambda *_args, **_kwargs: pytest.fail("runner must not be called"), + ) + + with pytest.raises(ValueError, match="declared post-clone source operator"): + run_multispine_post_clone_source_operator( + _source_frame(), + "with_us_housing_inputs", + ) + + +def test_source_output_merge_materializes_boolean_without_numeric_coercion() -> None: + pool = _source_frame() + operated = pool.select(np.asarray([True, False])) + operated_person = operated.table("person").copy() + operated_person["fixture_flag"] = pd.Series( + [True], + index=operated_person.index, + dtype=bool, + ) + operated = _replace_person(operated, operated_person) + outputs = {"person": frozenset({"fixture_flag"})} + + merged, rows = multispine_pool_module._merge_source_operator_outputs( + pool, + operated, + outputs, + operator_name="fixture_boolean", + ) + + flag = merged.table("person")["fixture_flag"] + assert rows == {"person": 1} + assert pd.api.types.is_bool_dtype(flag.dtype) + assert flag.tolist() == [True, pd.NA] + + incumbent_person = pool.table("person").copy() + incumbent_person["fixture_flag"] = pd.Series( + [pd.NA, False], + index=incumbent_person.index, + dtype=object, + ) + incumbent = _replace_person(pool, incumbent_person) + preserved, _rows = multispine_pool_module._merge_source_operator_outputs( + incumbent, + operated, + outputs, + operator_name="fixture_boolean", + ) + preserved_flag = preserved.table("person")["fixture_flag"] + assert pd.api.types.is_bool_dtype(preserved_flag.dtype) + assert preserved_flag.tolist() == [True, False] + + numeric_person = pool.table("person").copy() + numeric_person["fixture_flag"] = np.asarray([np.nan, 0.0]) + numeric = _replace_person(pool, numeric_person) + with pytest.raises( + TypeError, + match=( + r"fixture_boolean.*physical booleans for person\.fixture_flag.*" + r"observed non-boolean values.*float64.*builtins\.float" + ), + ): + multispine_pool_module._merge_source_operator_outputs( + numeric, + operated, + outputs, + operator_name="fixture_boolean", + ) + + +def test_puf_allocation_basis_maps_boolean_incidence_explicitly() -> None: + declared_boolean_bases = ( + puf_support_module._PERSON_OUTPUT_BOOLEAN_INCIDENCE_DISTRIBUTION_BASES + ) + assert declared_boolean_bases == { + ("qualified_tuition_expenses", "is_full_time_college_student"), + ("sstb_self_employment_income_before_lsr", "business_is_sstb"), + ("sstb_unadjusted_basis_qualified_property", "business_is_sstb"), + ("sstb_w2_wages_from_qualified_business", "business_is_sstb"), + } + metric_by_column = { + (entity, column): metric + for ( + entity, + _family, + column, + _clone_index, + ), metric in stacked_spine_module.CANONICAL_ORIGIN_BATTERY_METRIC_REGISTRY.items() + } + for output_column, basis_column in declared_boolean_bases: + assert metric_by_column[("person", output_column)] == ( + "monetary_sign_separated" + ) + assert metric_by_column[("person", basis_column)] == "boolean_incidence" + + boolean_basis = pd.Series([True, False, pd.NA], dtype="boolean") + np.testing.assert_array_equal( + puf_support_module._nonnegative_allocation_basis_values( + boolean_basis, + output_column="qualified_tuition_expenses", + basis_column="is_full_time_college_student", + ), + np.asarray([1.0, 0.0, 0.0]), + ) + object_boolean_basis = pd.Series( + [np.bool_(True), None, False], + dtype=object, + ) + np.testing.assert_array_equal( + puf_support_module._nonnegative_allocation_basis_values( + object_boolean_basis, + output_column="sstb_w2_wages_from_qualified_business", + basis_column="business_is_sstb", + ), + np.asarray([1.0, 0.0, 0.0]), + ) + np.testing.assert_array_equal( + puf_support_module._nonnegative_allocation_basis_values( + pd.Series([2.5, -1.0, pd.NA], dtype="Float64"), + output_column="qualified_tuition_expenses", + basis_column="fixture_amount", + ), + np.asarray([2.5, 0.0, 0.0]), + ) + + with pytest.raises( + TypeError, + match=( + r"qualified_tuition_expenses.*fixture_amount.*" + r"monetary_sign_separated.*real numeric values.*builtins\.str" + ), + ): + puf_support_module._nonnegative_allocation_basis_values( + pd.Series(["1.0"], dtype=object), + output_column="qualified_tuition_expenses", + basis_column="fixture_amount", + ) + with pytest.raises( + TypeError, + match=( + r"qualified_tuition_expenses.*fixture_amount.*" + r"monetary_sign_separated.*physical booleans" + ), + ): + puf_support_module._nonnegative_allocation_basis_values( + pd.Series([True, 1.0], dtype=object), + output_column="qualified_tuition_expenses", + basis_column="fixture_amount", + ) + with pytest.raises( + TypeError, + match=( + r"qualified_tuition_expenses.*is_full_time_college_student.*" + r"boolean_incidence.*physical booleans.*float64" + ), + ): + puf_support_module._nonnegative_allocation_basis_values( + pd.Series([0.0, 1.0], dtype=np.float64), + output_column="qualified_tuition_expenses", + basis_column="is_full_time_college_student", + ) + np.testing.assert_array_equal( + puf_support_module._nonnegative_allocation_basis_values( + pd.Series([0.0, 1.0, np.nan], dtype=np.float64), + output_column="qualified_tuition_expenses", + basis_column="is_full_time_college_student", + allow_legacy_numeric_boolean=True, + ), + np.asarray([0.0, 1.0, 0.0]), + ) + with pytest.raises( + ValueError, + match=r"legacy allocation basis.*boolean_incidence.*outside exact \{0, 1\}", + ): + puf_support_module._nonnegative_allocation_basis_values( + pd.Series([2.0], dtype=np.float64), + output_column="qualified_tuition_expenses", + basis_column="is_full_time_college_student", + allow_legacy_numeric_boolean=True, + ) + with pytest.raises( + ValueError, + match=r"qualified_tuition_expenses.*fixture_amount.*1 nonfinite", + ): + puf_support_module._nonnegative_allocation_basis_values( + pd.Series([np.inf], dtype=np.float64), + output_column="qualified_tuition_expenses", + basis_column="fixture_amount", + ) + + person = pd.DataFrame( + { + "person_tax_unit_id": [10, 10, 20, 20], + "qualified_tuition_expenses": [np.nan, np.nan, np.nan, np.nan], + "is_full_time_college_student": pd.Series( + [False, True, pd.NA, False], + dtype="boolean", + ), + } + ) + student_basis = person["is_full_time_college_student"].copy() + puf_support_module._write_person_tax_unit_totals( + person, + mask=pd.Series(True, index=person.index), + column="qualified_tuition_expenses", + totals=pd.Series({10: 100.0, 20: 50.0}), + nonnegative=True, + fallback_basis_columns=("is_full_time_college_student",), + ) + np.testing.assert_array_equal( + person["qualified_tuition_expenses"].to_numpy(), + np.asarray([0.0, 100.0, 50.0, 0.0]), + ) + pd.testing.assert_series_equal( + person["is_full_time_college_student"], + student_basis, + ) + + qbi_person = pd.DataFrame( + { + "person_tax_unit_id": [30, 30], + "sstb_w2_wages_from_qualified_business": [np.nan, np.nan], + "business_is_sstb": pd.Series([False, True], dtype="boolean"), + } + ) + qbi_basis = qbi_person["business_is_sstb"].copy() + puf_support_module._write_person_tax_unit_totals( + qbi_person, + mask=pd.Series(True, index=qbi_person.index), + column="sstb_w2_wages_from_qualified_business", + totals=pd.Series({30: 60.0}), + nonnegative=True, + fallback_basis_columns=("business_is_sstb",), + ) + np.testing.assert_array_equal( + qbi_person["sstb_w2_wages_from_qualified_business"].to_numpy(), + np.asarray([0.0, 60.0]), + ) + pd.testing.assert_series_equal(qbi_person["business_is_sstb"], qbi_basis) + + +def _single_post_clone_source_receipt(operator: str) -> dict[str, object]: + return { + "phase": "post_clone", + "operator_order": [operator], + "cps_source_evidence": {"column": "PERIDNUM", "person_rows": 4}, + "transient_outputs_carried_through_clone": {}, + "suboperators": [ + { + "operator": operator, + "order_index": 0, + "phase": "post_clone", + } + ], + } + + +def test_source_finalizer_requires_all_16_receipts_and_preserves_run_order( + monkeypatch: pytest.MonkeyPatch, +) -> None: + execution_order = tuple(reversed(POOL_POST_CLONE_SOURCE_OPERATOR_ORDER)) + receipts = { + operator: _single_post_clone_source_receipt(operator) + for operator in execution_order + } + deferred_calls: list[Frame] = [] + + def materialize_once(frame: Frame) -> PoolStageOutput: + deferred_calls.append(frame) + return PoolStageOutput(frame, {"inputs": {"fixture": {"status": "pending"}}}) + + monkeypatch.setattr( + multispine_pool_module, + "materialize_pool_deferred_transfer_inputs", + materialize_once, + ) + + finalized = finalize_multispine_source_inputs( + _source_frame(), + operator_receipts=receipts, + ) + + assert deferred_calls == [finalized.frame] + assert finalized.receipt["operator_order"] == list(execution_order) + assert [item["order_index"] for item in finalized.receipt["suboperators"]] == list( + range(16) + ) + assert finalized.receipt["deferred_transfer_inputs"] == { + "inputs": {"fixture": {"status": "pending"}} + } + + +def test_source_finalizer_rejects_incomplete_receipts_before_deferred_inputs( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr( + multispine_pool_module, + "materialize_pool_deferred_transfer_inputs", + lambda _frame: pytest.fail("deferred inputs must not be materialized"), + ) + receipts = { + operator: _single_post_clone_source_receipt(operator) + for operator in POOL_POST_CLONE_SOURCE_OPERATOR_ORDER[:-1] + } + + with pytest.raises(ValueError, match=r"exactly.*16.*missing=.*education"): + finalize_multispine_source_inputs( + _source_frame(), + operator_receipts=receipts, + ) + + +def test_source_finalizer_rejects_formula_owned_outputs_before_deferred_inputs( + monkeypatch: pytest.MonkeyPatch, +) -> None: + frame = _source_frame() + frame.table("person")["weeks_worked"] = 52.0 + receipts = { + operator: _single_post_clone_source_receipt(operator) + for operator in POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + } + monkeypatch.setattr( + multispine_pool_module, + "materialize_pool_deferred_transfer_inputs", + lambda _frame: pytest.fail("deferred inputs must not be materialized"), + ) + + with pytest.raises(ValueError, match="formula-owned source"): + finalize_multispine_source_inputs( + frame, + operator_receipts=receipts, + ) + + def test_production_operator_invocations_are_total_and_guarded( monkeypatch: pytest.MonkeyPatch, ) -> None: + pytest.importorskip("policyengine_us") structural_expectations = ( ( multispine_pool_module.prepare_multispine_puf_predictors, @@ -1947,14 +2868,9 @@ def test_production_operator_invocations_are_total_and_guarded( {"_run_source_operator_chain"}, ), ( - multispine_pool_module.complete_multispine_source_inputs, + multispine_pool_module._post_clone_source_operators, POOL_POST_CLONE_SOURCE_OPERATOR_ORDER, - { - "_assert_formula_owned_source_outputs_absent", - "_run_source_operator_chain", - "materialize_pool_deferred_transfer_inputs", - "PoolStageOutput", - }, + set(), ), ( multispine_pool_module.derive_multispine_pool_inputs, @@ -1965,6 +2881,7 @@ def test_production_operator_invocations_are_total_and_guarded( "bind_us_qbi_reconciliation_transition_authority", "dict", "list", + "pool_remaining_stage_input_manifest_receipt", "us_qbi_reconciliation_change_receipt", "validate_us_qbi_reconciliation_live_output", "validate_us_qbi_reconciliation_transition", @@ -2051,7 +2968,10 @@ def observe_guarded_chain( assert observed == [ ("pre_clone", POOL_PRE_CLONE_SOURCE_OPERATOR_ORDER), - ("post_clone", POOL_POST_CLONE_SOURCE_OPERATOR_ORDER), + *( + ("post_clone", (operator,)) + for operator in POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + ), ("post_clone", POOL_DERIVE_OPERATOR_ORDER), ] observed_placements = { @@ -2070,6 +2990,7 @@ def observe_guarded_chain( def test_derive_stage_rejects_preclone_pool_before_kernels( monkeypatch: pytest.MonkeyPatch, ) -> None: + pytest.importorskip("policyengine_us") assembled = assemble_spines( {"asec": _source_frame(), "acs": _source_frame()}, household_mass_shares={"asec": 0.5, "acs": 0.5}, @@ -2097,6 +3018,7 @@ def unexpected_kernel(frame: Frame) -> Frame: def test_derive_stage_keeps_whole_pool_qbi_reconciliation() -> None: + pytest.importorskip("policyengine_us") assembled = assemble_spines( {"asec": _source_frame(), "acs": _source_frame()}, household_mass_shares={"asec": 0.5, "acs": 0.5}, @@ -2116,6 +3038,12 @@ def test_derive_stage_keeps_whole_pool_qbi_reconciliation() -> None: derived = result.frame.table("person") assert result.receipt["operator_order"] == list(POOL_DERIVE_OPERATOR_ORDER) + assert result.receipt["remaining_stage_input_manifest"]["entry_count"] == 993 + assert result.receipt["remaining_stage_input_manifest"]["stage_counts"] == { + "derive": 34, + "seed": 29, + "simulate": 930, + } assert ( result.receipt["qbi_input_reconciliation"]["recipient_source_universe"][ "rows_excluded_from_base_self_employment_rewrite" @@ -2156,6 +3084,7 @@ def _qbi_ready_derive_frame() -> Frame: def test_derive_stage_rejects_forged_qbi_kernel_receipt( monkeypatch: pytest.MonkeyPatch, ) -> None: + pytest.importorskip("policyengine_us") monkeypatch.setattr( multispine_pool_module, "us_qbi_reconciliation_change_receipt", @@ -2174,6 +3103,7 @@ def test_derive_stage_rejects_forged_qbi_kernel_receipt( def test_derive_stage_rejects_mutated_qbi_output_with_fresh_receipt( monkeypatch: pytest.MonkeyPatch, ) -> None: + pytest.importorskip("policyengine_us") real_kernel = multispine_pool_module.with_us_qbi_input_reconciliation def mutate_kernel(frame: Frame) -> Frame: @@ -2828,6 +3758,31 @@ def test_pool_asset_deferrals_are_typed_null_receipted_and_fail_when_stale() -> materialize_pool_deferred_transfer_inputs(result.frame) +def test_pool_asset_deferrals_materialize_the_declared_physical_dtype( + monkeypatch: pytest.MonkeyPatch, +) -> None: + declarations = { + column: {**declaration, "physical_dtype": "float32"} + for column, declaration in POOL_DEFERRED_TRANSFER_INPUTS.items() + } + monkeypatch.setattr( + multispine_pool_module, + "POOL_DEFERRED_TRANSFER_INPUTS", + declarations, + ) + + result = materialize_pool_deferred_transfer_inputs( + _assembled_cloned_with_partial_take_up() + ) + + person = result.frame.table("person") + assert all(person[column].dtype == np.dtype("float32") for column in declarations) + assert all( + receipt["physical_dtype"] == "float32" + for receipt in result.receipt["inputs"].values() + ) + + def test_pool_seed_stage_preserves_inputs_and_receipts_disclosed_defaults() -> None: frame = _assembled_cloned_with_partial_take_up() before_person = frame.table("person") diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py b/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py index 73da327b..8f5dc55e 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py @@ -9,6 +9,7 @@ import pytest import microcosm.build.us_runtime.h5_io as h5_io +import microcosm.build.us_runtime.stacked_spine as stacked_spine_module from microcosm.build.frame_checkpoint import ( load_frame_checkpoint, write_frame_checkpoint, @@ -21,8 +22,10 @@ from microcosm.build.us_runtime.h5_io import ( US_MULTISPINE_AGREEMENT_DIAGNOSTICS_ARTIFACT_KIND, US_MULTISPINE_POOL_H5_ARTIFACT_KIND, + US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION, US_MULTISPINE_POOL_MANIFEST_ARTIFACT_KIND, US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION, + US_STACKED_POOL_OPERATOR_ORDER, AuthenticatedPoolH5MismatchError, load_simulation_ready_us_multispine_pool, write_nullable_us_h5, @@ -272,6 +275,26 @@ def test_pool_export_rejects_ambiguous_object_strings_before_replacement( assert output.read_bytes() == b"previous-good-pool" +@pytest.mark.parametrize("materializer_version", (True, 0, -1, 1.5, "2")) +def test_pool_export_rejects_invalid_materializer_versions_before_replacement( + tmp_path: Path, + materializer_version: object, +) -> None: + output = tmp_path / "existing.pool.h5" + output.write_bytes(b"previous-good-pool") + + with pytest.raises(ValueError, match="positive integer"): + write_nullable_us_h5( + _pool_frame(), + output, + period=2024, + artifact_kind=US_MULTISPINE_POOL_H5_ARTIFACT_KIND, + materializer_version=materializer_version, # type: ignore[arg-type] + ) + + assert output.read_bytes() == b"previous-good-pool" + + def test_pool_export_rejects_untyped_all_missing_objects_before_replacement( tmp_path: Path, ) -> None: @@ -377,16 +400,20 @@ def _write_ready_pool(tmp_path: Path, *, stacked: bool = False) -> Path: } }, } + schema_version = US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION if stacked else 4 write_nullable_us_h5( _pool_frame_with_object_strings_on_every_entity(), pool_path, period=2024, artifact_kind=US_MULTISPINE_POOL_H5_ARTIFACT_KIND, publication_run_id=run_id, + materializer_version=( + US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION if stacked else None + ), ) diagnostics = { "artifact_kind": (US_MULTISPINE_AGREEMENT_DIAGNOSTICS_ARTIFACT_KIND), - "schema_version": US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION, + "schema_version": schema_version, "simulation_ready": True, "publication_run_id": run_id, "agreement_gate": agreement_gate, @@ -395,23 +422,41 @@ def _write_ready_pool(tmp_path: Path, *, stacked: bool = False) -> Path: diagnostics.update( { "pipeline": "us-stacked-pool", + "semantic_kind": "stacked_terminal_gates", "terminal_gates": agreement_gate, } ) diagnostics_path.write_text(json.dumps(diagnostics), encoding="utf-8") manifest = { "artifact_kind": US_MULTISPINE_POOL_MANIFEST_ARTIFACT_KIND, - "schema_version": US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION, + "schema_version": schema_version, "status": "simulation_ready", "simulation_ready": True, "publication_run_id": run_id, "period": 2024, + "operator_order": [ + "assemble", + "clone", + "impute", + "derive", + "seed", + "simulate", + "agreement", + ], + "stage_receipts": { + stage: {"operator": stage} + for stage in ("impute", "derive", "seed", "simulate") + }, "stage_checkpoints": { + "artifact_kind": "populace_us_multispine_pool_checkpoint_provenance", + "schema_version": 1, + "materializer_version": 3 if not stacked else 5, + "enabled": False, "agreement": { "source": "always_fresh", "cached": False, "terminal_verdict_persisted": False, - } + }, }, "agreement_gate": agreement_gate, "provenance_counts": {"household": {"rows": 3}}, @@ -429,12 +474,32 @@ def _write_ready_pool(tmp_path: Path, *, stacked: bool = False) -> Path: }, } if stacked: + dag = _canonical_stacked_late_dag_receipt() + transition_authority = ( + stacked_spine_module._late_producer_transition_authority_receipt(dag) + ) manifest.update( { "pipeline": "us-stacked-pool", + "late_producer_transition_authority_sha256": ( + transition_authority["sha256"] + ), "terminal_gates": agreement_gate, + "operator_order": list(US_STACKED_POOL_OPERATOR_ORDER), + "stage_receipts": { + "impute": { + "source_operator_chain": { + "late_dag_completion": dag["source_completion"], + }, + "stacked_late_producer_dag": dag, + "stacked_post_puf_transfer": dag["post_puf_transfer"], + } + }, } ) + manifest["pool_h5"]["materializer_version"] = ( + US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION + ) manifest_path.write_text( json.dumps(manifest), encoding="utf-8", @@ -442,6 +507,296 @@ def _write_ready_pool(tmp_path: Path, *, stacked: bool = False) -> Path: return manifest_path +def _canonical_stacked_late_dag_receipt() -> dict[str, object]: + """Build a signed fixture receipt over the live canonical contracts.""" + + schedule = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_SCHEDULE + schedule_receipt = stacked_spine_module._json_ready( + stacked_spine_module.us_late_producer_schedule_receipt() + ) + source_order = [ + producer.removeprefix("source:") + for producer in schedule.order + if producer.startswith("source:") + ] + source_receipts = { + operator: { + "phase": "post_clone", + "operator_order": [operator], + "cps_source_evidence": None, + "suboperators": [{"operator": operator}], + } + for operator in source_order + } + source_completion = { + "phase": "post_clone", + "operator_order": source_order, + "cps_source_evidence": None, + "suboperators": [ + {"operator": operator, "order_index": index} + for index, operator in enumerate(source_order) + ], + "deferred_transfer_inputs": { + "inputs": { + column: {} + for column in ( + "bank_account_assets", + "bond_assets", + "stock_assets", + ) + } + }, + } + group_receipts = { + group.name: { + "producer": group.name, + "entity": group.entity, + "family": group.family, + "ordered_targets": list(group.targets), + "targets": { + f"{group.entity}/{group.family}/{target}": { + "residual_null_rows": 0, + } + for target in group.targets + }, + } + for group in stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS + } + group_by_name = { + group.name: group + for group in stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS + } + canonical_family = { + (entity, target): family + for entity, families in ( + stacked_spine_module.CANONICAL_STACKED_POST_PUF_TRANSFER_SURFACE.items() + ) + for family, targets in families.items() + for target in targets + } + aggregate_targets = { + f"{group.entity}/{canonical_family[(group.entity, target)]}/{target}": ( + group_receipts[group.name]["targets"][ + f"{group.entity}/{group.family}/{target}" + ] + ) + for group in stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS + for target in group.targets + } + transfer = { + "authority": dict(stacked_spine_module.stacked_spine_authority_receipt()), + "producer_schedule": schedule_receipt, + "producer_execution_order": [ + producer + for producer in schedule.order + if producer != stacked_spine_module.US_LATE_PRIMARY_PUF_STAGE + ], + "groups": group_receipts, + "targets": aggregate_targets, + "completion": { + "status": "complete", + "group_count": 19, + "target_count": 70, + "residual_null_rows": 0, + }, + } + input_frame_sha256 = "1" * 64 + previous_sha256 = stacked_spine_module._late_execution_genesis_sha256( + producer_schedule_sha256=schedule_receipt["payload_sha256"], + input_frame_sha256=input_frame_sha256, + ) + execution = [] + for index, producer_name in enumerate(schedule.order): + contract = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY[ + producer_name + ] + if contract.kind == "acs_earnings_universe": + available = ( + stacked_spine_module._late_acs_earnings_universe_resource_receipts() + ) + elif contract.kind == "primary_puf": + available = stacked_spine_module.stacked_late_primary_resource_receipts( + pd.DataFrame({"fixture_donor": [1.0]}), + primary_qrf_checkpoint_identity_sha256="5" * 64, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + seed=0, + n_estimators=100, + fit_records_enabled=True, + tail_bound_diagnostics_enabled=True, + ) + elif contract.kind == "post_clone_source": + available = stacked_spine_module._late_source_resource_receipts( + producer_name=producer_name, + ) + elif contract.kind == "source_finalizer": + available = { + f"person.@source_receipt:{operator}": ( + stacked_spine_module._late_available_input_receipt( + producer=producer_name, + entity="person", + column=f"@source_receipt:{operator}", + rows=1, + binding={ + "resource_kind": "source_operator_receipt", + "schema_version": 1, + "source_operator": operator, + "source_receipt_sha256": ( + stacked_spine_module._canonical_sha256(source_receipt) + ), + }, + ) + ) + for operator, source_receipt in source_receipts.items() + } + available.update( + stacked_spine_module._late_source_finalizer_resource_receipts() + ) + elif contract.kind == "late_transfer": + group = group_by_name[producer_name] + available = stacked_spine_module._late_transfer_resource_receipts( + group_name=group.name, + entity=group.entity, + family=group.family, + targets=group.targets, + seed=0, + n_estimators=100, + max_targets_per_fit=( + stacked_spine_module.DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT + ), + target_bank=None, + ) + else: + available = {} + declared_inputs = [] + for requirement in contract.inputs: + alternatives = [] + for alternative in requirement.alternatives: + physical_evidence = [] + for column in alternative: + is_virtual = ( + column.column.startswith("@") + and column.column != "@resolved_weight" + and column.entity != "frame" + ) + key = f"{column.entity}.{column.column}" + resource_receipt = available.get(key) if is_virtual else None + present = not is_virtual or resource_receipt is not None + physical_evidence.append( + { + "entity": column.entity, + "column": column.column, + "value_kind": column.value_kind, + "required_scope": requirement.required_scope, + "scope_rows": 1, + "missing_rows": 0 if present else 1, + "invalid_rows": 0, + "status": "present" if present else "absent", + "content_sha256": ( + stacked_spine_module._canonical_sha256(resource_receipt) + if resource_receipt is not None + else "2" * 64 + ), + **( + {"weight_kind": "household_weight"} + if column.column == "@resolved_weight" + else {} + ), + } + ) + alternatives.append(physical_evidence) + evidence = {"alternatives": alternatives} + evidence["sha256"] = stacked_spine_module._canonical_sha256(evidence) + declared_inputs.append( + { + "entity": requirement.entity, + "column": requirement.column, + "required_scope": requirement.required_scope, + "producing_stage": requirement.producing_stage, + "unfilled_rows": 0, + "invalid_rows": 0, + "evidence": evidence, + } + ) + output_surface = [ + { + "entity": output.entity, + "column": output.column, + "coverage_scope": output.coverage_scope, + "status": "present", + "content_sha256": "3" * 64, + **({} if output.entity == "frame" else {"scope_rows": 1}), + **( + {"weight_kind": "household_weight"} + if output.column == "@resolved_weight" + else {} + ), + } + for output in contract.outputs + ] + if contract.kind == "acs_earnings_universe": + producer_receipt = {"fixture": "acs_earnings_universe"} + elif contract.kind == "primary_puf": + producer_receipt = { + "primary_resource_receipts_sha256": ( + stacked_spine_module._canonical_sha256(available) + ) + } + elif contract.kind == "post_clone_source": + producer_receipt = source_receipts[producer_name.removeprefix("source:")] + elif contract.kind == "source_finalizer": + producer_receipt = source_completion + elif contract.kind == "late_transfer": + producer_receipt = group_receipts[group_by_name[producer_name].name] + else: + producer_receipt = {} + for output in output_surface: + if output["column"].startswith("@source_receipt:"): + output["content_sha256"] = stacked_spine_module._canonical_sha256( + producer_receipt + ) + row = { + "execution_index": index, + "producer": producer_name, + "kind": contract.kind, + "declared_inputs": declared_inputs, + "declared_absence_receipts": {}, + "available_input_receipts": available, + "input_surface_sha256": stacked_spine_module._canonical_sha256( + declared_inputs + ), + "output_surface": output_surface, + "output_surface_sha256": stacked_spine_module._canonical_sha256( + output_surface + ), + "producer_receipt": producer_receipt, + "producer_receipt_sha256": stacked_spine_module._canonical_sha256( + producer_receipt + ), + "previous_execution_sha256": previous_sha256, + "status": "complete", + } + row["sha256"] = stacked_spine_module._canonical_sha256(row) + previous_sha256 = row["sha256"] + execution.append(row) + receipt = { + "version": stacked_spine_module.US_LATE_PRODUCER_RECEIPT_SCHEMA_VERSION, + "producer_schedule": schedule_receipt, + "input_frame_sha256": input_frame_sha256, + "output_frame_sha256": "4" * 64, + "execution_chain_sha256": previous_sha256, + "execution": execution, + "source_completion": source_completion, + "post_puf_transfer": transfer, + } + receipt["sha256"] = stacked_spine_module._canonical_sha256(receipt) + stacked_spine_module.validate_stacked_late_producer_receipt( + receipt, + boundary="canonical stacked H5 fixture", + ) + return receipt + + def test_ready_pool_loader_preserves_importance_weights_and_nullable_inputs( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, @@ -494,6 +849,47 @@ def replace_after_pinned_read(path: Path) -> bytes: ) +def test_ready_legacy_pool_loader_accepts_pre_653_schema_four_envelope( + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool(tmp_path) + written_manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + diagnostics_path = Path(written_manifest["agreement_diagnostics"]["path"]) + written_diagnostics = json.loads(diagnostics_path.read_text(encoding="utf-8")) + + frame, loaded_manifest, _authenticated_h5 = ( + load_simulation_ready_us_multispine_pool(manifest_path) + ) + + assert written_manifest["schema_version"] == 4 + assert written_diagnostics["schema_version"] == 4 + assert loaded_manifest["schema_version"] == 4 + assert "materializer_version" not in written_manifest["pool_h5"] + assert "materializer_version" not in h5_io.read_nullable_us_h5_metadata( + written_manifest["pool_h5"]["path"] + ) + assert frame.n("household") == 3 + + +def test_ready_legacy_pool_loader_rejects_current_stacked_envelope( + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool(tmp_path) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + diagnostics_path = Path(manifest["agreement_diagnostics"]["path"]) + diagnostics = json.loads(diagnostics_path.read_text(encoding="utf-8")) + manifest["schema_version"] = US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION + diagnostics["schema_version"] = US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION + diagnostics_path.write_text(json.dumps(diagnostics), encoding="utf-8") + manifest["agreement_diagnostics"]["sha256"] = _sha256(diagnostics_path) + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(ValueError, match="ambiguous stacked envelope"): + load_simulation_ready_us_multispine_pool(manifest_path) + + def test_ready_pool_loader_rejects_a_false_h5_size_receipt(tmp_path: Path) -> None: pytest.importorskip("tables") manifest_path = _write_ready_pool(tmp_path) @@ -594,9 +990,179 @@ def test_ready_stacked_pool_loader_binds_terminal_gate_aliases( pytest.importorskip("tables") manifest_path = _write_ready_pool(tmp_path, stacked=True) - _, manifest, _ = load_simulation_ready_us_multispine_pool(manifest_path) + frame, manifest, _ = load_simulation_ready_us_multispine_pool(manifest_path) assert manifest["terminal_gates"] == manifest["agreement_gate"] + assert manifest["schema_version"] == 8 + assert ( + manifest["pool_h5"]["materializer_version"] + == US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION + ) + assert ( + h5_io.read_nullable_us_h5_metadata(manifest["pool_h5"]["path"])[ + "materializer_version" + ] + == US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION + ) + transition_authority = frame.metadata[ + stacked_spine_module.US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY + ] + assert ( + transition_authority["sha256"] + == manifest["late_producer_transition_authority_sha256"] + ) + + +@pytest.mark.parametrize("location", ("manifest", "h5")) +@pytest.mark.parametrize("value", (None, 1, True)) +def test_ready_stacked_pool_loader_requires_exact_h5_materializer_binding( + tmp_path: Path, + location: str, + value: object, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool(tmp_path, stacked=True) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + if location == "manifest": + if value is None: + del manifest["pool_h5"]["materializer_version"] + else: + manifest["pool_h5"]["materializer_version"] = value + else: + pool_path = Path(manifest["pool_h5"]["path"]) + with pd.HDFStore(pool_path, mode="a") as store: + metadata = json.loads(str(store["_populace_staging_metadata"].iloc[0])) + if value is None: + del metadata["materializer_version"] + else: + metadata["materializer_version"] = value + store.put( + "_populace_staging_metadata", + pd.Series([json.dumps(metadata, sort_keys=True)]), + format="table", + ) + manifest["pool_h5"]["sha256"] = _sha256(pool_path) + manifest["pool_h5"]["size_bytes"] = pool_path.stat().st_size + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(ValueError, match="current H5 materializer version"): + load_simulation_ready_us_multispine_pool(manifest_path) + + +def test_ready_stacked_pool_loader_requires_current_late_dag_proof( + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool(tmp_path, stacked=True) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + del manifest["stage_receipts"]["impute"]["stacked_late_producer_dag"] + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(ValueError, match="has no late-producer DAG receipt"): + load_simulation_ready_us_multispine_pool(manifest_path) + + +def test_ready_stacked_pool_loader_rejects_schema_four_envelope( + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool(tmp_path, stacked=True) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + diagnostics_path = Path(manifest["agreement_diagnostics"]["path"]) + diagnostics = json.loads(diagnostics_path.read_text(encoding="utf-8")) + manifest["schema_version"] = 4 + diagnostics["schema_version"] = 4 + diagnostics_path.write_text(json.dumps(diagnostics), encoding="utf-8") + manifest["agreement_diagnostics"]["sha256"] = _sha256(diagnostics_path) + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(ValueError, match="legacy envelope carries stacked-only"): + load_simulation_ready_us_multispine_pool(manifest_path) + + +def test_ready_stacked_pool_cannot_be_downgraded_to_legacy( + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool(tmp_path, stacked=True) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + diagnostics_path = Path(manifest["agreement_diagnostics"]["path"]) + diagnostics = json.loads(diagnostics_path.read_text(encoding="utf-8")) + manifest["schema_version"] = 4 + manifest.pop("pipeline") + manifest.pop("terminal_gates") + diagnostics["schema_version"] = 4 + diagnostics.pop("pipeline") + diagnostics.pop("semantic_kind") + diagnostics.pop("terminal_gates") + diagnostics_path.write_text(json.dumps(diagnostics), encoding="utf-8") + manifest["agreement_diagnostics"]["sha256"] = _sha256(diagnostics_path) + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(ValueError, match="legacy envelope carries stacked-only"): + load_simulation_ready_us_multispine_pool(manifest_path) + + +def test_ready_stacked_pool_cannot_be_stripped_into_legacy_shape( + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool(tmp_path, stacked=True) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + diagnostics_path = Path(manifest["agreement_diagnostics"]["path"]) + diagnostics = json.loads(diagnostics_path.read_text(encoding="utf-8")) + manifest["schema_version"] = 4 + for field in ( + "pipeline", + "release_id", + "sampling", + "clone_attachment", + "input_pins_digest", + "late_producer_transition_authority_sha256", + "stack_manifest", + "terminal_gates", + "operator_order", + "stage_receipts", + ): + manifest.pop(field, None) + diagnostics["schema_version"] = 4 + for field in ("pipeline", "semantic_kind", "release_id", "terminal_gates"): + diagnostics.pop(field, None) + diagnostics_path.write_text(json.dumps(diagnostics), encoding="utf-8") + manifest["agreement_diagnostics"]["sha256"] = _sha256(diagnostics_path) + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + # A lazy strip that misses the pool_h5 receipt's materializer_version is + # caught earliest, by the stacked-only-marker refusal. + with pytest.raises(ValueError, match="stacked-only field"): + load_simulation_ready_us_multispine_pool(manifest_path) + + # Even a complete strip must still refuse at the canonical-envelope check. + manifest["pool_h5"].pop("materializer_version", None) + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + with pytest.raises(ValueError, match="canonical legacy envelope"): + load_simulation_ready_us_multispine_pool(manifest_path) + + +@pytest.mark.parametrize("authority", [None, "0" * 64]) +def test_ready_stacked_pool_loader_rejects_late_authority_mismatch( + tmp_path: Path, + authority: str | None, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool(tmp_path, stacked=True) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + if authority is None: + del manifest["late_producer_transition_authority_sha256"] + else: + manifest["late_producer_transition_authority_sha256"] = authority + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises( + ValueError, + match="independently carried late-producer transition authority", + ): + load_simulation_ready_us_multispine_pool(manifest_path) @pytest.mark.parametrize("document", ["manifest", "diagnostics"]) diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py index 986b01e5..7aaf2bb3 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py @@ -512,6 +512,7 @@ def _run_checkpoint_fixture( target_bank_receipt: Mapping[str, object] | None = None, primary_qrf_manifest_path: Path | None = None, authenticated_qbi: bool = True, + checkpoint_nullable_booleans: bool = False, ): order: list[str] = [] @@ -523,6 +524,22 @@ def apply(frame: Frame) -> PoolStageOutput: order.append(name) person = frame.table("person").copy() transform(person) + if name == "impute" and checkpoint_nullable_booleans: + complete = np.resize( + np.asarray([True, False], dtype=np.bool_), + len(person), + ) + missing = pd.array(complete, dtype="boolean") + missing[1] = pd.NA + person["is_female"] = pd.Series( + complete, + index=person.index, + dtype="boolean", + ) + person["fixture_declared_boolean"] = pd.Series( + missing, + index=person.index, + ) receipt: dict[str, object] = {"fixture_stage": name} if name == "impute" and primary_qrf_manifest_path is not None: receipt = { @@ -843,7 +860,7 @@ def _assert_publication_tombstone( "publication_run_id": publication_run_id, }, "publication_run_id": publication_run_id, - "schema_version": pool_tool.POOL_MANIFEST_SCHEMA_VERSION, + "schema_version": pool_tool._LEGACY_POOL_MANIFEST_SCHEMA_VERSION, "simulation_ready": False, "status": "publication_in_progress", } @@ -896,6 +913,352 @@ def _noncanonical_post_puf_authority_receipt() -> dict[str, object]: return stacked_spine_module._authority_receipt(test_authority) +def _canonical_late_transfer_receipt( + pool_tool: ModuleType, + *, + authority: Mapping[str, object] | None = None, +) -> dict[str, object]: + canonical_family = { + (entity, target): family + for entity, families in ( + pool_tool.CANONICAL_STACKED_POST_PUF_TRANSFER_SURFACE.items() + ) + for family, targets in families.items() + for target in targets + } + groups: dict[str, object] = {} + targets: dict[str, object] = {} + for group in pool_tool.CANONICAL_US_LATE_TRANSFER_GROUPS: + group_targets = { + f"{group.entity}/{group.family}/{target}": {"residual_null_rows": 0} + for target in group.targets + } + groups[group.name] = { + "producer": group.name, + "ordered_targets": list(group.targets), + "targets": group_targets, + } + for target in group.targets: + targets[ + f"{group.entity}/{canonical_family[(group.entity, target)]}/{target}" + ] = group_targets[f"{group.entity}/{group.family}/{target}"] + return { + "fixture": "post_puf_transfer", + "authority": dict( + pool_tool.stacked_spine_authority_receipt() + if authority is None + else authority + ), + "producer_schedule": pool_tool._json_ready( + pool_tool.us_late_producer_schedule_receipt() + ), + "producer_execution_order": [ + producer + for producer in stacked_spine_module.CANONICAL_US_LATE_PRODUCER_SCHEDULE.order + if producer != stacked_spine_module.US_LATE_PRIMARY_PUF_STAGE + ], + "groups": groups, + "targets": targets, + "completion": { + "status": "complete", + "group_count": 19, + "target_count": 70, + "residual_null_rows": 0, + }, + } + + +def _canonical_late_dag_receipt( + pool_tool: ModuleType, + *, + authority: Mapping[str, object] | None = None, + output_frame_sha256: str = "f" * 64, +) -> dict[str, object]: + schedule = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_SCHEDULE + schedule_receipt = pool_tool._json_ready( + pool_tool.us_late_producer_schedule_receipt() + ) + source_order = [ + producer.removeprefix("source:") + for producer in schedule.order + if producer.startswith("source:") + ] + cps_source_evidence = {"fixture": "shared_cps_source_evidence"} + source_receipts = { + operator: { + "phase": "post_clone", + "operator_order": [operator], + "suboperators": [{"operator": operator, "order_index": 0}], + "cps_source_evidence": cps_source_evidence, + } + for operator in source_order + } + source_completion = { + "phase": "post_clone", + "operator_order": source_order, + "suboperators": [ + {"operator": operator, "order_index": index} + for index, operator in enumerate(source_order) + ], + "cps_source_evidence": cps_source_evidence, + "deferred_transfer_inputs": { + "inputs": { + column: {} + for column in ( + "bank_account_assets", + "bond_assets", + "stock_assets", + ) + } + }, + } + transfer = _canonical_late_transfer_receipt( + pool_tool, + authority=authority, + ) + input_frame_sha256 = "e" * 64 + previous_sha256 = stacked_spine_module._late_execution_genesis_sha256( + producer_schedule_sha256=schedule_receipt["payload_sha256"], + input_frame_sha256=input_frame_sha256, + ) + execution = [] + for index, producer_name in enumerate(schedule.order): + contract = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY[ + producer_name + ] + declared_inputs = [] + if contract.kind == "acs_earnings_universe": + available = ( + stacked_spine_module._late_acs_earnings_universe_resource_receipts() + ) + elif contract.kind == "primary_puf": + available: dict[str, object] = ( + stacked_spine_module.stacked_late_primary_resource_receipts( + pd.DataFrame({"fixture_donor": [1.0]}), + primary_qrf_checkpoint_identity_sha256="c" * 64, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + seed=0, + n_estimators=100, + fit_records_enabled=True, + tail_bound_diagnostics_enabled=True, + ) + ) + elif contract.kind == "post_clone_source": + available = stacked_spine_module._late_source_resource_receipts( + producer_name=producer_name, + ) + elif contract.kind == "source_finalizer": + available = { + f"person.@source_receipt:{operator}": ( + stacked_spine_module._late_available_input_receipt( + producer=producer_name, + entity="person", + column=f"@source_receipt:{operator}", + rows=1, + binding={ + "resource_kind": "source_operator_receipt", + "schema_version": 1, + "source_operator": operator, + "source_receipt_sha256": ( + stacked_spine_module._canonical_sha256( + source_receipts[operator] + ) + ), + }, + ) + ) + for operator in source_order + } + available.update( + stacked_spine_module._late_source_finalizer_resource_receipts() + ) + elif contract.kind == "late_transfer": + group = next( + group + for group in pool_tool.CANONICAL_US_LATE_TRANSFER_GROUPS + if group.name == producer_name + ) + available = stacked_spine_module._late_transfer_resource_receipts( + group_name=group.name, + entity=group.entity, + family=group.family, + targets=group.targets, + seed=0, + n_estimators=100, + max_targets_per_fit=( + stacked_spine_module.DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT + ), + target_bank=None, + ) + else: + available = {} + for item in contract.inputs: + alternatives = [] + for alternative in item.alternatives: + physical_evidence = [] + for column in alternative: + is_virtual = ( + column.column.startswith("@") + and column.column != "@resolved_weight" + and column.entity != "frame" + ) + key = f"{column.entity}.{column.column}" + resource_receipt = available.get(key) if is_virtual else None + present = not is_virtual or resource_receipt is not None + physical_evidence.append( + { + "entity": column.entity, + "column": column.column, + "value_kind": column.value_kind, + "required_scope": item.required_scope, + "scope_rows": 1, + "missing_rows": 0 if present else 1, + "invalid_rows": 0, + "status": "present" if present else "absent", + "content_sha256": ( + stacked_spine_module._canonical_sha256(resource_receipt) + if resource_receipt is not None + else "a" * 64 + ), + **( + {"weight_kind": "household_weight"} + if column.column == "@resolved_weight" + else {} + ), + } + ) + alternatives.append(physical_evidence) + evidence = {"alternatives": alternatives} + evidence["sha256"] = stacked_spine_module._canonical_sha256(evidence) + declared_inputs.append( + { + "entity": item.entity, + "column": item.column, + "required_scope": item.required_scope, + "producing_stage": item.producing_stage, + "unfilled_rows": 0, + "invalid_rows": 0, + "evidence": evidence, + } + ) + if contract.kind == "acs_earnings_universe": + producer_receipt = {"fixture": "acs_earnings_universe"} + elif contract.kind == "primary_puf": + producer_receipt: Mapping[str, object] = { + "fixture": "primary_puf", + "primary_resource_receipts_sha256": ( + stacked_spine_module._canonical_sha256(available) + ), + } + elif contract.kind == "post_clone_source": + producer_receipt = source_receipts[producer_name.removeprefix("source:")] + elif contract.kind == "source_finalizer": + producer_receipt = source_completion + else: + producer_receipt = transfer["groups"][producer_name] + output_surface = [ + { + "entity": output.entity, + "column": output.column, + "coverage_scope": output.coverage_scope, + "status": "present", + "content_sha256": ( + stacked_spine_module._canonical_sha256(producer_receipt) + if output.column.startswith("@source_receipt:") + else "b" * 64 + ), + **({} if output.entity == "frame" else {"scope_rows": 1}), + **( + {"weight_kind": "household_weight"} + if output.column == "@resolved_weight" + else {} + ), + } + for output in contract.outputs + ] + row: dict[str, object] = { + "execution_index": index, + "producer": producer_name, + "kind": contract.kind, + "declared_inputs": declared_inputs, + "declared_absence_receipts": {}, + "available_input_receipts": available, + "input_surface_sha256": stacked_spine_module._canonical_sha256( + declared_inputs + ), + "output_surface": output_surface, + "output_surface_sha256": stacked_spine_module._canonical_sha256( + output_surface + ), + "producer_receipt": producer_receipt, + "producer_receipt_sha256": stacked_spine_module._canonical_sha256( + producer_receipt + ), + "previous_execution_sha256": previous_sha256, + "status": "complete", + } + row["sha256"] = stacked_spine_module._canonical_sha256(row) + previous_sha256 = row["sha256"] + execution.append(row) + receipt: dict[str, object] = { + "version": stacked_spine_module.US_LATE_PRODUCER_RECEIPT_SCHEMA_VERSION, + "producer_schedule": schedule_receipt, + "input_frame_sha256": input_frame_sha256, + "output_frame_sha256": output_frame_sha256, + "execution_chain_sha256": previous_sha256, + "execution": execution, + "source_completion": source_completion, + "post_puf_transfer": transfer, + } + receipt["sha256"] = stacked_spine_module._canonical_sha256(receipt) + return receipt + + +def _canonical_late_impute_receipts( + pool_tool: ModuleType, + *, + authority: Mapping[str, object] | None = None, +) -> dict[str, object]: + dag = _canonical_late_dag_receipt(pool_tool, authority=authority) + return { + "source_operator_chain": { + "late_dag_completion": dag["source_completion"], + }, + "stacked_late_producer_dag": dag, + "stacked_post_puf_transfer": dag["post_puf_transfer"], + } + + +def _authorized_late_impute_fixture( + pool_tool: ModuleType, + frame: Frame, + *, + authority: Mapping[str, object] | None = None, +) -> tuple[Frame, dict[str, object], str]: + """Bind one structurally signed synthetic DAG proof to a live fixture frame.""" + + dag = _canonical_late_dag_receipt( + pool_tool, + authority=authority, + output_frame_sha256=stacked_spine_module._late_frame_content_sha256(frame), + ) + authorized, transition_authority_sha256 = ( + stacked_spine_module._bind_late_producer_transition_authority(frame, dag) + ) + return ( + authorized, + { + "source_operator_chain": { + "late_dag_completion": dag["source_completion"], + }, + "stacked_late_producer_dag": dag, + "stacked_post_puf_transfer": dag["post_puf_transfer"], + }, + transition_authority_sha256, + ) + + def _install_stacked_entrypoint_stubs( pool_tool: ModuleType, monkeypatch: pytest.MonkeyPatch, @@ -1024,12 +1387,20 @@ def gap_fill(frame: Frame, **kwargs): lambda _manifest: None, ) + observed_primary_qrf_binding: dict[str, object] = {} + def puf_pass(frame: Frame, donor: pd.DataFrame, **kwargs): order.append("puf") assert donor is puf_donor assert len(donor) == 7 assert kwargs["clone_attachment_fraction"] == 1.0 assert kwargs["clone_attachment_seed"] == 579 + primary_binding = kwargs["primary_qrf_input_binding"] + stacked_spine_module._validate_stacked_late_primary_checkpoint_input_binding( + primary_binding, + boundary="tool wiring fixture", + ) + observed_primary_qrf_binding.update(primary_binding) if terminal == "error": raise RuntimeError("fixture stacked error") checkpoint_dir = Path(kwargs["primary_qrf_checkpoint_dir"]) @@ -1046,37 +1417,91 @@ def puf_pass(frame: Frame, donor: pd.DataFrame, **kwargs): }, "puf_capital_gains_tail_transfer": {"fixture": "tail"}, "tail_status": "applied", + "primary_resource_receipts_sha256": ( + stacked_spine_module._canonical_sha256( + primary_binding["primary_resource_receipts"] + ) + ), }, ) monkeypatch.setattr(pool_tool, "run_stacked_puf_pass", puf_pass) - def complete(frame: Frame): - order.append("complete") - return PoolStageOutput(frame, {"fixture": "complete"}) - - monkeypatch.setattr(pool_tool, "complete_multispine_source_inputs", complete) - - def post_puf_transfer(frame: Frame, **kwargs: object): - order.append("post_puf_transfer") - assert kwargs["target_bank"] is not None + def late_producer_dag(frame: Frame, **kwargs: object): + primary_puf_result = kwargs["primary_puf_producer"](frame) + order.append("late_producer_dag") + assert set(kwargs["primary_resource_receipts"]) == { + "tax_unit.@puf_donor_tax_units", + "tax_unit.@primary_qrf_checkpoint", + "tax_unit.@primary_puf_execution_config", + } + assert ( + observed_primary_qrf_binding["primary_resource_receipts"] + == kwargs["primary_resource_receipts"] + ) + primary_config = kwargs["primary_resource_receipts"][ + "tax_unit.@primary_puf_execution_config" + ]["binding"] + assert primary_config["clone_attachment"] == { + "fraction": 1.0, + "seed": 579, + "support_channels": ["asec", "puf_tax_detail"], + "puf_clone_index": 1, + } + assert primary_config["qrf"]["seed"] == pool_tool.POOL_RANDOM_SEED + assert ( + primary_config["qrf"]["n_estimators"] == pool_tool._PRIMARY_QRF_N_ESTIMATORS + ) + target_banks = kwargs["target_banks"] + assert isinstance(target_banks, Mapping) + assert set(target_banks) == { + group.name for group in pool_tool.CANONICAL_US_LATE_TRANSFER_GROUPS + } + schedule_sha256 = pool_tool.us_late_producer_schedule_receipt()[ + "payload_sha256" + ] + dag_sha256 = pool_tool.us_late_producer_schedule_receipt()["schedule_sha256"] + for group in pool_tool.CANONICAL_US_LATE_TRANSFER_GROUPS: + bank = target_banks[group.name] + assert bank.root.parts[-3:] == ( + "late_producer_dag", + group.entity, + group.family, + ) + assert bank._identity["late_producer_dag_sha256"] == dag_sha256 + assert bank._identity["late_producer_schedule_sha256"] == schedule_sha256 + assert bank._identity["late_producer"] == { + "name": group.name, + "entity": group.entity, + "family": group.family, + "ordered_targets": list(group.targets), + } + dag_receipt = _canonical_late_dag_receipt( + pool_tool, + authority=post_puf_authority, + output_frame_sha256=stacked_spine_module._late_frame_content_sha256( + primary_puf_result.frame + ), + ) + authorized_frame, transition_authority_sha256 = ( + stacked_spine_module._bind_late_producer_transition_authority( + primary_puf_result.frame, + dag_receipt, + ) + ) return SimpleNamespace( - frame=frame, - receipt={ - "fixture": "post_puf_transfer", - "authority": dict( - pool_tool.stacked_spine_authority_receipt() - if post_puf_authority is None - else post_puf_authority - ), - }, + frame=authorized_frame, + receipt=dag_receipt, + primary_puf_result=primary_puf_result, + source_completion_receipt=dag_receipt["source_completion"], transfer_result=SimpleNamespace(fit_records=()), + transition_authority_sha256=transition_authority_sha256, ) monkeypatch.setattr( pool_tool, - "transfer_stacked_post_puf_inputs", - post_puf_transfer, + "run_stacked_late_producer_dag", + late_producer_dag, ) monkeypatch.setattr( pool_tool, @@ -1122,12 +1547,22 @@ def simulate(frame: Frame): simulate, ) - def completeness(_frame: Frame) -> GateResult: + def completeness( + _frame: Frame, + *, + tail_manifest: Mapping[str, object], + ) -> GateResult: order.append("completeness") + assert tail_manifest == {"fixture": "tail"} return GateResult(name="fixture_completeness", passed=True) - def battery(_frame: Frame) -> GateResult: + def battery( + _frame: Frame, + *, + tail_manifest: Mapping[str, object], + ) -> GateResult: order.append("battery") + assert tail_manifest == {"fixture": "tail"} if terminal == "red": return GateResult( name="fixture_battery", @@ -1245,8 +1680,7 @@ def test_stacked_tool_entrypoint_fixture_e2e_emits_one_logbook_row_at_every_term "prepare", "gap", "puf", - "complete", - "post_puf_transfer", + "late_producer_dag", "tail_prepare", "derive", "seed", @@ -1271,14 +1705,57 @@ def test_stacked_tool_entrypoint_fixture_e2e_emits_one_logbook_row_at_every_term manifest = json.loads( (tmp_path / "stacked-pool.manifest.json").read_text(encoding="utf-8") ) + assert manifest["schema_version"] == pool_tool.POOL_MANIFEST_SCHEMA_VERSION assert manifest["pipeline"] == "us-stacked-pool" + assert manifest["pool_h5"]["materializer_version"] == ( + pool_tool.US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION + ) + with pd.HDFStore(manifest["pool_h5"]["path"], mode="r") as store: + h5_metadata = json.loads(str(store["_populace_staging_metadata"].iloc[0])) + assert h5_metadata["materializer_version"] == ( + pool_tool.US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION + ) + published_dag = manifest["stage_receipts"]["impute"][ + "stacked_late_producer_dag" + ] + assert published_dag == _canonical_late_dag_receipt( + pool_tool, + output_frame_sha256=published_dag["output_frame_sha256"], + ) + expected_late_authority_sha256 = ( + stacked_spine_module._late_producer_transition_authority_receipt( + published_dag + )["sha256"] + ) + assert manifest["late_producer_transition_authority_sha256"] == ( + expected_late_authority_sha256 + ) + checkpoint_root = next( + (tmp_path / "stacked-pool.checkpoints" / "stacked").iterdir() + ) + for stage in ("transferred", "simulated"): + checkpoint_path = checkpoint_root / f"{stage}.checkpoint.h5" + checkpoint_manifest = json.loads( + checkpoint_path.with_suffix(".manifest.json").read_text( + encoding="utf-8" + ) + ) + checkpoint_metadata = pool_tool.load_frame_checkpoint( + checkpoint_path + ).metadata + assert ( + checkpoint_manifest["late_producer_transition_authority_sha256"] + == expected_late_authority_sha256 + ) + assert ( + checkpoint_metadata["late_producer_transition_authority_sha256"] + == expected_late_authority_sha256 + ) assert manifest["operator_order"] == [ "assemble_stacked_spine", "prepare_multispine_source_inputs_for_clone", "gap_fill_stacked_spine", - "run_stacked_puf_pass", - "complete_multispine_source_inputs", - "transfer_stacked_post_puf_inputs", + "run_stacked_late_producer_dag", "prepare_stacked_tail_derivation", "derive_multispine_pool_inputs", "seed_multispine_pool_inputs", @@ -1315,7 +1792,7 @@ def test_stacked_entrypoint_rejects_noncanonical_post_puf_transfer_receipt( with pytest.raises( ValueError, match=( - "stacked cold-build post-PUF transfer: non-canonical stacked " + "stacked cold-build late-producer DAG: non-canonical stacked " "authority is forbidden" ), ): @@ -1327,25 +1804,63 @@ def test_stacked_entrypoint_rejects_noncanonical_post_puf_transfer_receipt( "prepare", "gap", "puf", - "complete", - "post_puf_transfer", + "late_producer_dag", ] assert not (tmp_path / "stacked-pool.h5").exists() assert not (tmp_path / "stacked-pool.manifest.json").exists() +def test_stacked_checkpoint_emission_propagates_and_authenticates_late_authority( + pool_tool: ModuleType, +) -> None: + authorized, impute, transition_authority_sha256 = _authorized_late_impute_fixture( + pool_tool, _source_frame() + ) + captured: list[MultispinePoolCheckpoint] = [] + + pool_tool._emit_stacked_checkpoint( + captured.append, + stage="transferred", + frame=authorized, + assembly_receipt={}, + stage_receipts={"impute": impute}, + late_producer_transition_authority_sha256=(transition_authority_sha256), + ) + + assert len(captured) == 1 + assert captured[0].late_producer_transition_authority_sha256 == ( + transition_authority_sha256 + ) + with pytest.raises( + ValueError, + match="differs from the independently carried late-producer", + ): + pool_tool._emit_stacked_checkpoint( + captured.append, + stage="transferred", + frame=authorized, + assembly_receipt={}, + stage_receipts={"impute": impute}, + late_producer_transition_authority_sha256="0" * 64, + ) + assert len(captured) == 1 + + def test_stacked_publication_rejects_noncanonical_receipt_before_any_write( pool_tool: ModuleType, tmp_path: Path, ) -> None: noncanonical = _noncanonical_post_puf_authority_receipt() outputs = pool_tool._stacked_output_paths(tmp_path / "stacked-pool.h5") + authorized, impute, transition_authority_sha256 = _authorized_late_impute_fixture( + pool_tool, + _source_frame(), + authority=noncanonical, + ) result = SimpleNamespace( - stage_receipts={ - "impute": { - "stacked_post_puf_transfer": {"authority": noncanonical}, - } - } + frame=authorized, + stage_receipts={"impute": impute}, + late_producer_transition_authority_sha256=transition_authority_sha256, ) with pytest.raises( @@ -1372,6 +1887,97 @@ def test_stacked_publication_rejects_noncanonical_receipt_before_any_write( assert not outputs.agreement_diagnostics.exists() +def test_stacked_publication_rejects_forged_late_transition_authority( + pool_tool: ModuleType, + tmp_path: Path, +) -> None: + authorized, impute, _transition_authority_sha256 = _authorized_late_impute_fixture( + pool_tool, _source_frame() + ) + outputs = pool_tool._stacked_output_paths(tmp_path / "stacked-pool.h5") + result = SimpleNamespace( + frame=authorized, + stage_receipts={"impute": impute}, + late_producer_transition_authority_sha256="0" * 64, + ) + + with pytest.raises( + ValueError, + match="differs from the independently carried late-producer", + ): + pool_tool._write_stacked_outputs( + result, + outputs=outputs, + verified_inputs={}, + acs_source_manifest=pool_tool.load_acs_source_manifest(), + input_receipts={}, + checkpoint_provenance={}, + sample_fraction=0.01, + sample_seed=578, + clone_attachment_fraction=1.0, + clone_attachment_seed=579, + ) + + assert not outputs.pool_h5.exists() + assert not outputs.manifest.exists() + assert not outputs.agreement_diagnostics.exists() + + +def test_late_dag_validator_rejects_forged_execution_row( + pool_tool: ModuleType, +) -> None: + receipt = _canonical_late_dag_receipt(pool_tool) + receipt["execution"][1]["producer"] = "source:forged" + + with pytest.raises( + ValueError, + match=r"execution row 1 is misbound", + ): + pool_tool.validate_stacked_late_producer_receipt( + receipt, + boundary="forged execution regression", + ) + + +def test_stacked_publication_rejects_forged_derived_order_before_any_write( + pool_tool: ModuleType, + tmp_path: Path, +) -> None: + authorized, impute, transition_authority_sha256 = _authorized_late_impute_fixture( + pool_tool, _source_frame() + ) + impute["stacked_late_producer_dag"]["post_puf_transfer"][ + "producer_execution_order" + ] = ["forged:wrong"] + outputs = pool_tool._stacked_output_paths(tmp_path / "stacked-pool.h5") + result = SimpleNamespace( + frame=authorized, + stage_receipts={"impute": impute}, + late_producer_transition_authority_sha256=transition_authority_sha256, + ) + + with pytest.raises( + ValueError, + match=r"execution order does not match the derived late-producer schedule", + ): + pool_tool._write_stacked_outputs( + result, + outputs=outputs, + verified_inputs={}, + acs_source_manifest=pool_tool.load_acs_source_manifest(), + input_receipts={}, + checkpoint_provenance={}, + sample_fraction=0.01, + sample_seed=578, + clone_attachment_fraction=1.0, + clone_attachment_seed=579, + ) + + assert not outputs.pool_h5.exists() + assert not outputs.manifest.exists() + assert not outputs.agreement_diagnostics.exists() + + @pytest.mark.parametrize( "failure", ("negative_seed", "oversized_seed", "invalid_output", "code_pin"), @@ -1451,6 +2057,7 @@ def test_publication_error_keeps_gate_receipts_and_does_not_claim_stale_h5( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, ) -> None: + pytest.importorskip("policyengine_us") _order, _full_puf_rows = _install_stacked_entrypoint_stubs( pool_tool, monkeypatch, @@ -1504,7 +2111,7 @@ def test_logbook_gate_receipts_are_immutable_across_later_attempts( monkeypatch.setattr( pool_tool, "by_origin_battery", - lambda _frame: GateResult( + lambda _frame, *, tail_manifest: GateResult( name="fixture_battery", passed=False, failures=("later red verdict",), @@ -1542,6 +2149,7 @@ def test_stacked_checkpoint_identity_binds_both_scale_controls_and_manifest( pool_tool: ModuleType, tmp_path: Path, ) -> None: + pytest.importorskip("policyengine_us") verified = _verified_inputs_fixture(pool_tool, tmp_path / "pins") asec = _many_household_source_frame() acs = _many_household_source_frame(measured_offset=1_000.0) @@ -1681,12 +2289,81 @@ def identity( assert changed_store.load_deepest() is None -def test_stacked_checkpoint_identity_binds_v6_semantic_contracts( +def test_pool_checkpoint_identity_binds_late_producer_schedule( + pool_tool: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + verified = _verified_inputs_fixture(pool_tool, tmp_path / "pins") + + def identity() -> dict[str, object]: + return pool_tool._pool_checkpoint_base_identity( + verified, + policyengine_us_version="fixture-engine", + ) + + current = identity() + expected_schedule = pool_tool._json_ready( + pool_tool.us_late_producer_schedule_receipt() + ) + assert current["pool_code"]["late_producer_schedule"] == expected_schedule + + changed_schedule = copy.deepcopy(expected_schedule) + changed_schedule["payload_sha256"] = "0" * 64 + monkeypatch.setattr( + pool_tool, + "us_late_producer_schedule_receipt", + lambda: changed_schedule, + ) + changed = identity() + + assert pool_tool._pool_checkpoint_identity_sha256(changed) != ( + pool_tool._pool_checkpoint_identity_sha256(current) + ) + + +def test_legacy_checkpoint_identity_excludes_stacked_late_producer_schedule( + pool_tool: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + verified = _verified_inputs_fixture(pool_tool, tmp_path / "pins") + current = pool_tool._legacy_pool_checkpoint_base_identity( + verified, + policyengine_us_version="fixture-engine", + ) + assert current["materializer_version"] == 3 + assert "late_producer_schedule" not in current["pool_code"] + + changed_schedule = pool_tool._json_ready( + pool_tool.us_late_producer_schedule_receipt() + ) + changed_schedule["payload_sha256"] = "0" * 64 + monkeypatch.setattr( + pool_tool, + "us_late_producer_schedule_receipt", + lambda: changed_schedule, + ) + changed = pool_tool._legacy_pool_checkpoint_base_identity( + verified, + policyengine_us_version="fixture-engine", + ) + + assert changed == current + + +def test_stacked_checkpoint_identity_binds_v11_semantic_contracts( pool_tool: ModuleType, monkeypatch: pytest.MonkeyPatch, tmp_path: Path, capsys: pytest.CaptureFixture[str], ) -> None: + pytest.importorskip("policyengine_us") + monkeypatch.setattr( + pool_tool, + "_policyengine_us_version", + lambda: "fixture-engine", + ) verified = _verified_inputs_fixture(pool_tool, tmp_path / "pins") stack = pool_tool.assemble_stacked_spine( _many_household_source_frame(), @@ -1708,15 +2385,57 @@ def identity() -> dict[str, object]: current = identity() pool_code = current["pool_code"] - assert current["materializer_version"] == 6 - assert current["stacked_authority"]["version"] == 6 + assert current["materializer_version"] == 11 + assert current["stacked_authority"]["version"] == 10 + assert pool_code["operator_order"] == [ + "assemble_stacked_spine", + "prepare_multispine_source_inputs_for_clone", + "gap_fill_stacked_spine", + "run_stacked_late_producer_dag", + "prepare_stacked_tail_derivation", + "derive_multispine_pool_inputs", + "seed_multispine_pool_inputs", + "materialize_multispine_agreement_outputs", + "stacked_completeness_gate", + "by_origin_battery", + ] + assert pool_code["late_producer_schedule"] == pool_tool._json_ready( + pool_tool.us_late_producer_schedule_receipt() + ) + resource_semantics = pool_code["late_producer_resource_semantics"] + unsigned_resource_semantics = dict(resource_semantics) + resource_semantics_sha256 = unsigned_resource_semantics.pop("sha256") + assert resource_semantics_sha256 == stacked_spine_module._canonical_sha256( + unsigned_resource_semantics + ) + assert resource_semantics["producer_count"] == 38 + resource_rows = { + row["producer"]: row["resources"] for row in resource_semantics["producers"] + } + assert list(resource_rows) == list( + stacked_spine_module.CANONICAL_US_LATE_PRODUCER_SCHEDULE.order + ) + for ( + producer, + contract, + ) in stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY.items(): + assert set(resource_rows[producer]) == ( + stacked_spine_module._late_contract_available_input_keys(contract) + ) assert pool_code["primary_qrf_checkpoint_schema_version"] == 6 + assert pool_code["puf_capital_gains_tail_manifest_schema_version"] == 2 + assert pool_code["puf_capital_gains_tail_support_contract"] == ( + pool_tool.puf_capital_gains_tail_support_contract_identity() + ) assert pool_code["acs_pums_earnings_universe_contract"] == ( pool_tool.acs_pums_earnings_universe_contract_identity() ) assert pool_code["us_qbi_reconciliation_contract"] == ( pool_tool.us_qbi_reconciliation_contract_identity() ) + assert pool_code["remaining_stage_input_manifest"] == ( + pool_tool.pool_remaining_stage_input_manifest_receipt() + ) with monkeypatch.context() as changed: changed.setattr(pool_tool, "PRIMARY_QRF_CHECKPOINT_SCHEMA_VERSION", 5) @@ -1748,12 +2467,108 @@ def identity() -> dict[str, object]: lambda: qbi_contract, ) stale_qbi = identity() + with monkeypatch.context() as changed: + changed.setattr(pool_tool, "PUF_CAPITAL_GAINS_TAIL_MANIFEST_SCHEMA_VERSION", 1) + stale_tail_schema = identity() + with monkeypatch.context() as changed: + remaining_manifest = copy.deepcopy( + pool_tool.pool_remaining_stage_input_manifest_receipt() + ) + remaining_manifest["manifest_sha256"] = "0" * 64 + changed.setattr( + pool_tool, + "pool_remaining_stage_input_manifest_receipt", + lambda: remaining_manifest, + ) + stale_remaining_manifest = identity() + with monkeypatch.context() as changed: + tail_contract = copy.deepcopy( + pool_tool.puf_capital_gains_tail_support_contract_identity() + ) + tail_contract["required_minimum"] = "one_recipient_per_status" + changed.setattr( + pool_tool, + "puf_capital_gains_tail_support_contract_identity", + lambda: tail_contract, + ) + stale_tail_contract = identity() + with monkeypatch.context() as changed: + late_schedule = pool_tool._json_ready( + pool_tool.us_late_producer_schedule_receipt() + ) + late_schedule["payload_sha256"] = "0" * 64 + changed.setattr( + pool_tool, + "us_late_producer_schedule_receipt", + lambda: late_schedule, + ) + stale_late_schedule = identity() + with monkeypatch.context() as changed: + source_stage_binding = stacked_spine_module._late_source_stage_spec_binding + + def changed_source_stage_binding( + operator: str, + **kwargs: object, + ) -> dict[str, object] | None: + binding = source_stage_binding(operator, **kwargs) + if operator != "with_us_adult_care_inputs" or binding is None: + return binding + mutated = copy.deepcopy(binding) + mutated["asset_sha256"] = "0" * 64 + return mutated + + changed.setattr( + stacked_spine_module, + "_late_source_stage_spec_binding", + changed_source_stage_binding, + ) + stale_source_asset = identity() digests = { pool_tool._pool_checkpoint_identity_sha256(candidate) - for candidate in (current, stale_qrf, stale_acs, stale_qbi) + for candidate in ( + current, + stale_qrf, + stale_acs, + stale_qbi, + stale_tail_schema, + stale_remaining_manifest, + stale_tail_contract, + stale_late_schedule, + stale_source_asset, + ) } - assert len(digests) == 4 + assert len(digests) == 9 + + # Positive control: discovery accepts the exact current semantic identity + # under the same fixture engine version used to construct it. + current_checkpoint_root = tmp_path / "current-semantic-checkpoints" + current_store = pool_tool._PoolStageCheckpointStore( + current_checkpoint_root, + base_identity=current, + ) + current_store.bind_input_receipts(_checkpoint_fixture_input_receipts()) + current_store.write( + pool_tool.MultispinePoolCheckpoint( + stage="assembled", + frame=stack.frame, + assembly_receipt=stack.frame.metadata[ + pool_tool.SPINE_ASSEMBLY_MANIFEST_KEY + ], + stage_receipts={}, + ) + ) + assert ( + pool_tool._discover_stacked_checkpoint_identity( + current_checkpoint_root, + verified_inputs=verified, + sample_fraction=0.10, + sample_seed=578, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + ) + == current + ) # A checkpoint produced by the current materializer with the prior QRF # schema is not merely identity-distinct: discovery must refuse it as stale. @@ -1774,7 +2589,7 @@ def identity() -> dict[str, object]: ) ) - assert current["materializer_version"] == stale_qrf["materializer_version"] == 6 + assert current["materializer_version"] == stale_qrf["materializer_version"] == 11 assert stale_qrf["pool_code"]["primary_qrf_checkpoint_schema_version"] == 5 assert ( pool_tool._discover_stacked_checkpoint_identity( @@ -1789,6 +2604,103 @@ def identity() -> dict[str, object]: ) assert "checkpoint base identity is stale" in capsys.readouterr().out + # Resource semantics are equally resume-fatal: a checkpoint whose source + # asset/config binding differs must never be selected under current code. + resource_checkpoint_root = tmp_path / "mixed-resource-checkpoints" + stale_resource_store = pool_tool._PoolStageCheckpointStore( + resource_checkpoint_root, + base_identity=stale_source_asset, + ) + stale_resource_store.bind_input_receipts(_checkpoint_fixture_input_receipts()) + stale_resource_store.write( + pool_tool.MultispinePoolCheckpoint( + stage="assembled", + frame=stack.frame, + assembly_receipt=stack.frame.metadata[ + pool_tool.SPINE_ASSEMBLY_MANIFEST_KEY + ], + stage_receipts={}, + ) + ) + + assert ( + pool_tool._discover_stacked_checkpoint_identity( + resource_checkpoint_root, + verified_inputs=verified, + sample_fraction=0.10, + sample_seed=578, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + ) + is None + ) + assert "checkpoint base identity is stale" in capsys.readouterr().out + + +def test_pool_envelope_v7_preserves_stacked_bank_identity_but_rejects_v6( + pool_tool: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + pytest.importorskip("policyengine_us") + verified = _verified_inputs_fixture(pool_tool, tmp_path / "pins") + stack = pool_tool.assemble_stacked_spine( + _many_household_source_frame(), + _many_household_source_frame(measured_offset=1_000.0), + sample_fraction=0.10, + sample_seed=578, + ) + + def identity() -> dict[str, object]: + return pool_tool._stacked_checkpoint_base_identity( + verified, + stack_receipt=stack.receipt, + sample_fraction=0.10, + sample_seed=578, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + policyengine_us_version="fixture-engine", + ) + + current_identity = identity() + current_digest = pool_tool._pool_checkpoint_identity_sha256(current_identity) + checkpoint_root = tmp_path / "envelope-version-checkpoints" + with monkeypatch.context() as legacy: + legacy.setattr( + pool_tool, + "POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION", + 6, + ) + assert identity() == current_identity + legacy_store = pool_tool._PoolStageCheckpointStore( + checkpoint_root, + base_identity=current_identity, + ) + assert legacy_store.base_identity_sha256 == current_digest + legacy_store.bind_input_receipts(_checkpoint_fixture_input_receipts()) + legacy_store.write( + pool_tool.MultispinePoolCheckpoint( + stage="assembled", + frame=stack.frame, + assembly_receipt=stack.frame.metadata[ + pool_tool.SPINE_ASSEMBLY_MANIFEST_KEY + ], + stage_receipts={}, + ) + ) + capsys.readouterr() + + assert pool_tool.POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION == 7 + assert identity() == current_identity + current_store = pool_tool._PoolStageCheckpointStore( + checkpoint_root, + base_identity=current_identity, + ) + assert current_store.base_identity_sha256 == current_digest + assert current_store.load_deepest() is None + assert "unsupported binding" in capsys.readouterr().out + @pytest.mark.parametrize( ("route", "stage_receipts"), @@ -1868,7 +2780,7 @@ def test_qbi_receipt_route_resolution_rejects_wrong_or_ambiguous_paths( ) -@pytest.mark.parametrize("legacy_version", (1, 2, 3, 4, 5)) +@pytest.mark.parametrize("legacy_version", (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)) def test_legacy_stacked_materializer_checkpoint_is_not_discovered( pool_tool: ModuleType, monkeypatch: pytest.MonkeyPatch, @@ -1876,6 +2788,12 @@ def test_legacy_stacked_materializer_checkpoint_is_not_discovered( capsys: pytest.CaptureFixture[str], legacy_version: int, ) -> None: + pytest.importorskip("policyengine_us") + monkeypatch.setattr( + pool_tool, + "_policyengine_us_version", + lambda: "fixture-engine", + ) verified = _verified_inputs_fixture(pool_tool, tmp_path / "pins") stack = pool_tool.assemble_stacked_spine( _many_household_source_frame(), @@ -1916,7 +2834,7 @@ def test_legacy_stacked_materializer_checkpoint_is_not_discovered( ) ) - assert pool_tool._STACKED_CHECKPOINT_MATERIALIZER_VERSION == 6 + assert pool_tool._STACKED_CHECKPOINT_MATERIALIZER_VERSION == 11 assert ( pool_tool._discover_stacked_checkpoint_identity( checkpoint_root, @@ -1942,15 +2860,17 @@ def test_stacked_resume_rejects_noncanonical_post_puf_transfer_receipt( sample_seed=578, ) noncanonical = _noncanonical_post_puf_authority_receipt() + authorized, impute, transition_authority_sha256 = _authorized_late_impute_fixture( + pool_tool, + stack.frame, + authority=noncanonical, + ) resume = pool_tool.MultispinePoolCheckpoint( stage="transferred", - frame=stack.frame, + frame=authorized, assembly_receipt=stack.frame.metadata[pool_tool.SPINE_ASSEMBLY_MANIFEST_KEY], - stage_receipts={ - "impute": { - "stacked_post_puf_transfer": {"authority": noncanonical}, - } - }, + stage_receipts={"impute": impute}, + late_producer_transition_authority_sha256=transition_authority_sha256, ) with pytest.raises( @@ -2041,8 +2961,7 @@ def test_stacked_entrypoint_resumes_each_checkpoint_boundary( "prepare", "gap", "puf", - "complete", - "post_puf_transfer", + "late_producer_dag", "tail_prepare", "derive", "seed", @@ -2072,8 +2991,7 @@ def test_stacked_entrypoint_resumes_each_checkpoint_boundary( "prepare", "gap", "puf", - "complete", - "post_puf_transfer", + "late_producer_dag", "tail_prepare", "derive", "seed", @@ -2305,8 +3223,23 @@ def deterministic_fixture_h5( assert keywords["source_native_inputs"] == {"acs": loaded.acs_native_inputs} assert keywords["resume"] is None assert callable(keywords["checkpoint"]) + checkpoint_store = keywords["checkpoint"].__self__ + assert checkpoint_store.base_identity["materializer_version"] == 3 + assert "late_producer_schedule" not in checkpoint_store.base_identity["pool_code"] outputs = pool_tool._output_paths(output, checkpoint_root=checkpoint_root) + manifest = pool_tool._read_json_object(outputs.manifest) + diagnostics = pool_tool._read_json_object(outputs.agreement_diagnostics) + assert pool_tool.POOL_MANIFEST_SCHEMA_VERSION == 8 + assert pool_tool.POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION == 7 + assert manifest["schema_version"] == 4 + assert diagnostics["schema_version"] == 4 + assert "materializer_version" not in manifest["pool_h5"] + assert manifest["stage_checkpoints"]["materializer_version"] == 3 + assert { + receipt["materializer_version"] + for receipt in manifest["stage_checkpoints"]["stages"].values() + } == {3} manifest_bytes = outputs.manifest.read_bytes().replace( str(tmp_path.resolve()).encode(), b"$TMP", @@ -2326,7 +3259,9 @@ def deterministic_fixture_h5( # checkpoint metadata). "pool_h5": "ced797ecdd44a638c2a3945f07ad612098a7095ca53a5f458699bca6d6e38b3e", "agreement": "f39f0d918bf7ee01dddb5517d8830b8adb541273c5be084307be91397caca3cb", - "manifest": "94604e72e589675f89013d9d8eb9518abd32832d1ff53045fb7ae06e48c1b146", + # Exact pre-#653 schema-4/materializer-3 publication bytes from + # preserved #652 commit 54d2dee6. + "manifest": "14e6b3a409dfe2108253668a65ed32c0365b246f379ad895d8441c939adde65e", } @@ -3424,11 +4359,60 @@ def test_pool_checkpoint_round_trip_resumes_each_boundary_byte_identically( } -def test_simulated_v2_checkpoint_accepts_both_string_encodings_without_rewrite( +def test_pool_checkpoint_store_round_trips_nullable_boolean_families( pool_tool: ModuleType, tmp_path: Path, ) -> None: - """V2 authenticates both physical string encodings as one logical frame.""" + checkpoint_root = tmp_path / "nullable-boolean-checkpoints" + cold_store = _checkpoint_fixture_store(pool_tool, checkpoint_root) + cold_store.bind_input_receipts(_checkpoint_fixture_input_receipts()) + + _run_checkpoint_fixture( + pool_tool, + tmp_path, + store=cold_store, + checkpoint_nullable_booleans=True, + ) + + h5py = pytest.importorskip("h5py") + for stage, expected_schema in ( + ("assembled", 2), + ("transferred", 3), + ("simulated", 3), + ): + path = cold_store.checkpoint_path(stage) + with h5py.File(path, mode="r") as h5: + raw = np.asarray(h5["_populace_frame_checkpoint/metadata_json"]).tobytes() + assert json.loads(raw)["schema_version"] == expected_schema + manifest = pool_tool._read_json_object( + cold_store.checkpoint_manifest_path(stage) + ) + assert manifest["materializer_version"] == 7 + loaded = pool_tool.load_frame_checkpoint(path).frame + if stage == "assembled": + assert "fixture_declared_boolean" not in loaded.person + continue + assert loaded.person["is_female"].dtype == pd.BooleanDtype() + assert loaded.person["fixture_declared_boolean"].dtype == pd.BooleanDtype() + assert not loaded.person["is_female"].isna().any() + assert loaded.person["fixture_declared_boolean"].isna().sum() == 1 + + cold_store.checkpoint_path("simulated").unlink() + cold_store.checkpoint_manifest_path("simulated").unlink() + warm_store = _checkpoint_fixture_store(pool_tool, checkpoint_root) + resumed = warm_store.load_deepest() + + assert resumed is not None + assert resumed.stage == "transferred" + assert resumed.frame.person["is_female"].dtype == pd.BooleanDtype() + assert resumed.frame.person["fixture_declared_boolean"].isna().sum() == 1 + + +def test_simulated_v7_checkpoint_accepts_both_string_encodings_without_rewrite( + pool_tool: ModuleType, + tmp_path: Path, +) -> None: + """V7 authenticates both physical string encodings as one logical frame.""" pytest.importorskip("h5py") checkpoint_root = tmp_path / "checkpoints" @@ -3440,7 +4424,7 @@ def test_simulated_v2_checkpoint_accepts_both_string_encodings_without_rewrite( loaded = pool_tool.load_frame_checkpoint(checkpoint_path) canonical_v2_bytes = checkpoint_path.read_bytes() canonical_identity = loaded.metadata["identity"] - assert loaded.metadata["materializer_version"] == 2 + assert loaded.metadata["materializer_version"] == 7 assert any( column["dtype"] == str(CANONICAL_STRING_DTYPE) for columns in loaded.metadata["frame_schema"]["entities"].values() @@ -3471,7 +4455,7 @@ def test_simulated_v2_checkpoint_accepts_both_string_encodings_without_rewrite( banked_v2_bytes = checkpoint_path.read_bytes() assert banked_v2_bytes != canonical_v2_bytes assert legacy_metadata["identity"] == canonical_identity - assert legacy_metadata["materializer_version"] == 2 + assert legacy_metadata["materializer_version"] == 7 assert any( column["dtype"] == "object" for columns in legacy_metadata["frame_schema"]["entities"].values() @@ -3812,18 +4796,58 @@ def test_take_up_contract_identity_mutation_rebuilds_every_pool_boundary( assert provenance["stages"][stage]["load_status"] == "identity_mismatch" -def test_pool_materializer_v1_artifacts_fail_closed_with_named_receipts( +def test_tail_support_contract_identity_mutation_rebuilds_pool_checkpoints( + pool_tool: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + checkpoint_root = tmp_path / "tail-support-contract-checkpoints" + original = pool_tool.puf_capital_gains_tail_support_contract_identity() + cold_store = _checkpoint_fixture_store(pool_tool, checkpoint_root) + assert ( + cold_store.base_identity["pool_code"]["puf_capital_gains_tail_support_contract"] + == original + ) + cold_store.bind_input_receipts(_checkpoint_fixture_input_receipts()) + _run_checkpoint_fixture(pool_tool, tmp_path, store=cold_store) + + changed = copy.deepcopy(original) + changed["insufficient_support_action"] = "silently_widen" + monkeypatch.setattr( + pool_tool, + "puf_capital_gains_tail_support_contract_identity", + lambda: changed, + ) + changed_store = _checkpoint_fixture_store(pool_tool, checkpoint_root) + + assert ( + changed_store.base_identity["pool_code"][ + "puf_capital_gains_tail_support_contract" + ] + == changed + ) + assert changed_store.base_identity_sha256 != cold_store.base_identity_sha256 + assert changed_store.load_deepest() is None + + +@pytest.mark.parametrize("legacy_version", (1, 2, 3, 4, 5, 6)) +def test_legacy_pool_materializer_artifacts_fail_closed_with_named_receipts( pool_tool: ModuleType, monkeypatch: pytest.MonkeyPatch, tmp_path: Path, capsys: pytest.CaptureFixture[str], + legacy_version: int, ) -> None: - checkpoint_root = tmp_path / "materializer-v1-checkpoints" + checkpoint_root = tmp_path / "legacy-materializer-checkpoints" with monkeypatch.context() as legacy: - legacy.setattr(pool_tool, "POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION", 1) + legacy.setattr( + pool_tool, + "POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION", + legacy_version, + ) legacy_store = _checkpoint_fixture_store(pool_tool, checkpoint_root) - assert legacy_store.base_identity["materializer_version"] == 1 + assert legacy_store.base_identity["materializer_version"] == legacy_version legacy_store.bind_input_receipts(_checkpoint_fixture_input_receipts()) _run_checkpoint_fixture(pool_tool, tmp_path, store=legacy_store) for stage in pool_tool.POOL_CHECKPOINT_STAGE_ORDER: @@ -3833,15 +4857,15 @@ def test_pool_materializer_v1_artifacts_fail_closed_with_named_receipts( manifest = pool_tool._read_json_object( legacy_store.checkpoint_manifest_path(stage) ) - assert metadata["materializer_version"] == 1 - assert metadata["identity"]["materializer_version"] == 1 - assert manifest["materializer_version"] == 1 - assert manifest["identity"]["materializer_version"] == 1 + assert metadata["materializer_version"] == legacy_version + assert metadata["identity"]["materializer_version"] == legacy_version + assert manifest["materializer_version"] == legacy_version + assert manifest["identity"]["materializer_version"] == legacy_version capsys.readouterr() - assert pool_tool.POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION == 2 + assert pool_tool.POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION == 7 current_store = _checkpoint_fixture_store(pool_tool, checkpoint_root) - assert current_store.base_identity["materializer_version"] == 2 + assert current_store.base_identity["materializer_version"] == 7 assert current_store.load_deepest() is None output = capsys.readouterr().out @@ -4277,15 +5301,15 @@ def test_stacked_manifest_and_publication_reject_forged_qbi_receipt( ) receipt = copy.deepcopy(legacy.stage_receipts["derive"]["qbi_input_reconciliation"]) receipt["sha256"] = "0" * 64 + authorized, impute, transition_authority_sha256 = _authorized_late_impute_fixture( + pool_tool, legacy.frame + ) stacked = SimpleNamespace( - frame=legacy.frame, + frame=authorized, qbi_transition_authority_sha256=(legacy.qbi_transition_authority_sha256), + late_producer_transition_authority_sha256=transition_authority_sha256, stage_receipts={ - "impute": { - "stacked_post_puf_transfer": { - "authority": dict(pool_tool.stacked_spine_authority_receipt()) - } - }, + "impute": impute, "derive": {"pool_derivation": {"qbi_input_reconciliation": receipt}}, }, ) @@ -4444,6 +5468,7 @@ def test_red_outputs_preserve_receipts_and_exclude_simulation_output( assert "ssi" not in store["person"].columns metadata = json.loads(str(store["_populace_staging_metadata"].iloc[0])) assert metadata["publication_run_id"] == manifest["publication_run_id"] + assert "materializer_version" not in metadata def test_ready_reader_binds_manifest_h5_and_diagnostics_to_one_run( diff --git a/packages/microcosm-build/tests/test_us_puf_capital_gains_tail.py b/packages/microcosm-build/tests/test_us_puf_capital_gains_tail.py index 8ec24406..f2fb44a0 100644 --- a/packages/microcosm-build/tests/test_us_puf_capital_gains_tail.py +++ b/packages/microcosm-build/tests/test_us_puf_capital_gains_tail.py @@ -6,6 +6,7 @@ import hashlib import importlib.util import json +import pickle from pathlib import Path from types import SimpleNamespace @@ -14,6 +15,7 @@ import pytest import microcosm.build.us_runtime.puf_capital_gains_tail as tail_module +import microcosm.build.us_runtime.puf_interest_components as interest_module from microcosm.build.us_runtime.capital_gain_distributions import ( load_capital_gain_distribution_shares, ) @@ -128,6 +130,54 @@ def _expanded_recipient_frame() -> Frame: return clone_us_frame_for_puf_support(base) +def test_tail_execution_identity_binds_resolved_spec_and_soi_asset( + tmp_path: Path, +) -> None: + baseline = tail_module.puf_capital_gains_tail_execution_inputs_identity() + buckets = baseline["aggregate_disaggregation_spec"]["buckets"] + assert [bucket["recid"] for bucket in buckets] == sorted( + bucket["recid"] for bucket in buckets + ) + + source = interest_module.files("microcosm.build.us").joinpath( + interest_module._SOURCE_ASSET + ) + payload = json.loads(source.read_text(encoding="utf-8")) + payload["agi_bands"][0]["total_interest_paid_amount"] += 1 + payload["agi_bands"][0]["investment_interest_amount"] += 1 + changed_asset = tmp_path / interest_module._SOURCE_ASSET + changed_asset.write_text(json.dumps(payload), encoding="utf-8") + changed = interest_module.puf_e19200_interest_components_asset_identity( + changed_asset + ) + + soi = baseline["soi_e19200_agi_bands"] + assert soi["asset_sha256"] != changed["asset_sha256"] + assert soi["agi_bands"][0] != changed["agi_bands"][0] + assert soi["runtime_agi_bands"]["agi_bands"] == soi["agi_bands"] + assert len(soi["runtime_agi_bands"]["sha256"]) == 64 + + +def test_tail_execution_identity_rejects_runtime_soi_band_asset_drift( + monkeypatch: pytest.MonkeyPatch, +) -> None: + changed_first = dataclasses.replace( + tail_module.US_PUF_E19200_AGI_BANDS[0], + label="runtime-drift", + ) + monkeypatch.setattr( + tail_module, + "US_PUF_E19200_AGI_BANDS", + (changed_first, *tail_module.US_PUF_E19200_AGI_BANDS[1:]), + ) + + with pytest.raises( + ValueError, + match="runtime SOI AGI bands differ.*content-bound packaged asset", + ): + tail_module.puf_capital_gains_tail_execution_inputs_identity() + + def _donor() -> pd.DataFrame: return pd.DataFrame( { @@ -206,6 +256,47 @@ def _replace_entity_table(frame: Frame, entity: str, table: pd.DataFrame) -> Fra ) +def _frame_digest(frame: Frame) -> str: + """Hash every frame byte-bearing surface for pre-fix parity checks.""" + + payload = ( + [(entity, frame.table(entity)) for entity in frame.entities], + [ + ( + entity, + frame.weights_for(entity).values, + frame.weights_for(entity).kind.value, + ) + for entity in frame.weighted_entities + ], + frame.strata, + frame.mass_log, + ) + return hashlib.sha256(pickle.dumps(payload, protocol=5)).hexdigest() + + +def _pre_652_all_adequate_reference_frame() -> Frame: + """Run the pre-support-filter allocation path in the live test runtime.""" + + frame = _expanded_recipient_frame() + donor = _donor() + tail, _selection = select_puf_capital_gains_tail_donors(donor) + normalization = frame.weights_for("household").total / float(donor["weight"].sum()) + assigned_weights = tail["weight"].to_numpy(dtype=np.float64) * normalization + candidates = tail_module._recipient_candidates( + frame, + maximum_transfer_weight=float(assigned_weights.max()), + seed=567, + ) + assignments = tail_module._assign_tail_donors( + tail, + assigned_weights=assigned_weights, + candidates=candidates, + ) + reference, _clone_receipt = tail_module._clone_and_transfer(frame, assignments) + return reference + + def _load_support_builder_module(): root = Path(__file__).resolve().parents[3] path = root / "tools" / "build_us_puf_support_base.py" @@ -318,6 +409,12 @@ def test_tail_transfer_splits_weights_and_copies_joint_vectors( tmp_path: Path, ) -> None: frame = _expanded_recipient_frame() + person = frame.table("person").copy() + clone_index = person[support_clone_index_column("person")] + qbi_flag = pd.Series(pd.NA, index=person.index, dtype="boolean") + qbi_flag.loc[clone_index.eq(1)] = [True, False, True, False] + person["business_is_sstb"] = qbi_flag + frame = _replace_entity_table(frame, "person", person) donor = _donor() before_household_weights = frame.weights_for("household") before_employment_mass = float( @@ -412,6 +509,25 @@ def test_tail_transfer_splits_weights_and_copies_joint_vectors( person = transferred.table("person") tax_unit = transferred.table("tax_unit") household = transferred.table("household") + transferred_clone_index = person[support_clone_index_column("person")] + transferred_flag = person["business_is_sstb"] + assert pd.api.types.is_bool_dtype(transferred_flag.dtype) + assert transferred_flag.loc[transferred_clone_index.eq(0)].isna().all() + assert transferred_flag.loc[transferred_clone_index.eq(1)].notna().all() + source_id_column = support_source_id_column("person") + detail_flag_by_source = pd.Series( + transferred_flag.loc[transferred_clone_index.eq(1)].array, + index=person.loc[transferred_clone_index.eq(1), source_id_column], + ) + tail_flag = transferred_flag.loc[transferred_clone_index.eq(2)] + expected_tail_flag = person.loc[ + transferred_clone_index.eq(2), source_id_column + ].map(detail_flag_by_source) + pd.testing.assert_series_equal( + tail_flag.reset_index(drop=True), + expected_tail_flag.reset_index(drop=True), + check_names=False, + ) for record in manifest["records"]: tail_tax_unit_id = record["tail_tax_unit_id"] tail_household_id = record["tail_household_id"] @@ -519,6 +635,81 @@ def test_tail_transfer_splits_weights_and_copies_joint_vectors( write_puf_capital_gains_tail_manifest(manifest_path, tampered) +def test_thin_filing_status_is_named_counted_and_not_attached() -> None: + """A thin status is skipped whole while an adequate peer still attaches.""" + + frame = _expanded_recipient_frame() + donor = _donor() + donor.loc[donor["tax_unit_id"].eq(20), "filing_status_code"] = 3.0 + + transferred, manifest = transfer_puf_capital_gains_tail( + frame, + donor, + seed=567, + ) + + support = manifest["recipient_support"] + by_status = {receipt["filing_status"]: receipt for receipt in support["strata"]} + assert support["insufficient_support_stratum_count"] == 1 + assert support["insufficient_support_strata"] == ["SEPARATE"] + assert by_status["SEPARATE"] == { + "filing_status_code": 3, + "filing_status": "SEPARATE", + "status": "insufficient_support", + "observed_count": 0, + "required_minimum": 1, + "attached_donor_count": 0, + "skipped_donor_count": 1, + } + assert by_status["SINGLE"]["status"] == "attached" + assert by_status["SINGLE"]["observed_count"] == 2 + assert by_status["SINGLE"]["required_minimum"] == 1 + assert by_status["SURVIVING_SPOUSE"]["status"] == "not_applicable" + assert manifest["record_count"] == 1 + assert {record["donor_filing_status"] for record in manifest["records"]} == { + "SINGLE" + } + assert transferred.n("household") == frame.n("household") + 1 + + +def test_adequate_strata_match_pre_fix_frame_bytes() -> None: + """All-adequate fixtures preserve the exact pre-#652 allocation bytes.""" + + transferred, manifest = transfer_puf_capital_gains_tail( + _expanded_recipient_frame(), + _donor(), + seed=567, + ) + + # Pandas' pickle bytes vary across supported runtime versions, so compare + # against the exact pre-#652 path under the same runtime instead of blessing + # one environment's pickle digest. + assert _frame_digest(transferred) == _frame_digest( + _pre_652_all_adequate_reference_frame() + ) + assert manifest["assignment_sha256"] == ( + "1b2262da65fa851e0a990ca9f04dee661de0145724f82aef679557bc92418937" + ) + assert manifest["recipient_support"]["insufficient_support_strata"] == [] + + +def test_tail_support_receipt_tampering_fails_closed() -> None: + """Rehashing only the envelope cannot launder a changed support count.""" + + _transferred, manifest = transfer_puf_capital_gains_tail( + _expanded_recipient_frame(), + _donor(), + seed=567, + ) + tampered = json.loads(json.dumps(manifest)) + tampered["recipient_support"]["strata"][0]["observed_count"] += 1 + tampered.pop("manifest_sha256") + tampered["manifest_sha256"] = tail_module._canonical_sha256(tampered) + + with pytest.raises(ValueError, match="recipient-support SHA mismatch"): + tail_module.validate_puf_capital_gains_tail_manifest(tampered) + + def test_tail_transfer_rejects_group_membership_crossing_households() -> None: frame = _expanded_recipient_frame() tables = {entity: frame.table(entity).copy() for entity in frame.entities} diff --git a/packages/microcosm-build/tests/test_us_puf_support_base_builder.py b/packages/microcosm-build/tests/test_us_puf_support_base_builder.py index 47c67c04..7f2d44dd 100644 --- a/packages/microcosm-build/tests/test_us_puf_support_base_builder.py +++ b/packages/microcosm-build/tests/test_us_puf_support_base_builder.py @@ -14,6 +14,11 @@ US_PUF_SUPPORT_FIT_NAME, clone_us_frame_for_puf_support, ) +from microcosm.build.us_runtime.puf_capital_gains_tail import ( + PUF_CAPITAL_GAINS_TAIL_MANIFEST_SCHEMA_VERSION, + PUF_CAPITAL_GAINS_TAIL_STAGE_NAME, + puf_capital_gains_tail_support_contract_identity, +) from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights @@ -38,6 +43,105 @@ def _canonical_sha256(value: object) -> str: ).hexdigest() +def _valid_capital_gains_tail_manifest() -> dict[str, object]: + """Return the smallest schema-current manifest for repair-path tests.""" + + record = { + "donor_source_id": 1, + "donor_weight": 1.0, + "assigned_weight": 1.0, + "donor_filing_status_code": 1, + "donor_filing_status": "SINGLE", + "donor_agi_band_index": 0, + "donor_agi_band": "fixture", + "donor_is_synthetic": False, + "joint_vector": {}, + "recipient_household_source_id": 1, + "recipient_tax_unit_source_id": 1, + "recipient_household_id": 1, + "recipient_tax_unit_id": 1, + "tail_household_id": 2, + "tail_tax_unit_id": 2, + "tail_person_id": 2, + } + records = [record] + strata = [ + { + "filing_status_code": code, + "filing_status": name, + "status": "attached" if code == 1 else "not_applicable", + "observed_count": 1 if code == 1 else 0, + "required_minimum": 1 if code == 1 else 0, + "attached_donor_count": 1 if code == 1 else 0, + "skipped_donor_count": 0, + } + for code, name in ( + (1, "SINGLE"), + (2, "JOINT"), + (3, "SEPARATE"), + (4, "HEAD_OF_HOUSEHOLD"), + (5, "SURVIVING_SPOUSE"), + ) + ] + recipient_support: dict[str, object] = { + "contract": puf_capital_gains_tail_support_contract_identity(), + "candidate_count": 1, + "selected_donor_count": 1, + "attached_donor_count": 1, + "skipped_donor_count": 0, + "attached_stratum_count": 1, + "insufficient_support_stratum_count": 0, + "not_applicable_stratum_count": 4, + "insufficient_support_strata": [], + "strata": strata, + } + recipient_support["sha256"] = _canonical_sha256(recipient_support) + donor_projection = [ + { + key: record[key] + for key in ( + "donor_source_id", + "donor_weight", + "donor_filing_status_code", + "donor_filing_status", + "donor_agi_band_index", + "donor_agi_band", + "donor_is_synthetic", + "joint_vector", + ) + } + ] + assignment_projection = [ + { + key: record[key] + for key in ( + "donor_source_id", + "assigned_weight", + "recipient_household_source_id", + "recipient_tax_unit_source_id", + "recipient_household_id", + "recipient_tax_unit_id", + "tail_household_id", + "tail_tax_unit_id", + "tail_person_id", + ) + } + ] + manifest: dict[str, object] = { + "artifact_kind": "populace_puf_capital_gains_tail_transfer", + "schema_version": PUF_CAPITAL_GAINS_TAIL_MANIFEST_SCHEMA_VERSION, + "stage": PUF_CAPITAL_GAINS_TAIL_STAGE_NAME, + "boundary": {"tail_record_count": 1}, + "recipient_support": recipient_support, + "donor_records_sha256": _canonical_sha256(donor_projection), + "assignment_sha256": _canonical_sha256(assignment_projection), + "record_count": 1, + "records": records, + } + manifest["manifest_sha256"] = _canonical_sha256(manifest) + return manifest + + def _minimal_us_frame() -> Frame: person = pd.DataFrame( { @@ -500,14 +604,7 @@ def test_capital_gains_tail_manifest_repairs_from_checkpoint_copy( tmp_path: Path, ) -> None: builder = _load_support_builder_module() - records: list[dict[str, object]] = [] - manifest = { - "donor_records_sha256": _canonical_sha256(records), - "assignment_sha256": _canonical_sha256(records), - "record_count": 0, - "records": records, - } - manifest["manifest_sha256"] = _canonical_sha256(manifest) + manifest = _valid_capital_gains_tail_manifest() output = tmp_path / "out" / "tail.json" checkpoint = tmp_path / "checkpoints" / "artifacts" / "tail.json" file_sha256 = builder.write_puf_capital_gains_tail_manifest( @@ -525,7 +622,7 @@ def test_capital_gains_tail_manifest_repairs_from_checkpoint_copy( "manifest_sha256": manifest["manifest_sha256"], "donor_records_sha256": manifest["donor_records_sha256"], "assignment_sha256": manifest["assignment_sha256"], - "record_count": 0, + "record_count": 1, } if damage == "missing": output.unlink() diff --git a/packages/microcosm-build/tests/test_us_qbi_inputs.py b/packages/microcosm-build/tests/test_us_qbi_inputs.py index 06b0ddd8..5317ba6e 100644 --- a/packages/microcosm-build/tests/test_us_qbi_inputs.py +++ b/packages/microcosm-build/tests/test_us_qbi_inputs.py @@ -675,6 +675,24 @@ def test_reconciliation_rejects_wrong_schema_and_missing_inputs() -> None: ) +@pytest.mark.parametrize( + "consumer", + (with_us_qbi_input_reconciliation, us_qbi_inputs_summary), + ids=("reconciliation", "summary"), +) +def test_present_s_corp_column_retains_exact_whole_pool_nonfinite_check( + consumer, +) -> None: + person = _qbi_person(7) + person["s_corp_income"] = np.nan + + with pytest.raises( + ValueError, + match=r"US QBI input 's_corp_income' contains 7 nonfinite value\(s\)\.", + ): + consumer(_frame(person)) + + def test_summary_does_not_mutate_source_frame() -> None: reconciled = with_us_qbi_input_reconciliation(_frame(_qbi_person())) before = reconciled.table("person").copy(deep=True) diff --git a/packages/microcosm-build/tests/test_us_spine_blindness.py b/packages/microcosm-build/tests/test_us_spine_blindness.py index f4237030..dc807da7 100644 --- a/packages/microcosm-build/tests/test_us_spine_blindness.py +++ b/packages/microcosm-build/tests/test_us_spine_blindness.py @@ -71,10 +71,13 @@ _SPINE_BLIND_BUILD_TOOLS = (_REPOSITORY_ROOT / "tools" / "build_us_multispine_pool.py",) _REQUIRED_POOL_RUNTIME_MODULES = frozenset( { + "late_producer_dag.py", "multispine_pool.py", "puf_support.py", "spine_agreement.py", "spine_assembly.py", + "us_late_overlap_ownership.py", + "us_late_producer_registry.py", } ) _RETIRED_LATE_ASSEMBLY_MODULES = frozenset( @@ -103,6 +106,9 @@ # by-origin battery are origin-aware by charter. "stacked_spine.py", "support_provenance.py", # Centralized provenance compatibility. + # Declares provenance-scoped inputs and edges; never mutates rows. + "us_late_overlap_ownership.py", + "us_late_producer_registry.py", "warm_start_selection.py", # Provenance reporting and recovery. } ) @@ -203,6 +209,7 @@ "immigration.py", "input_mass.py", "l0_refit_export.py", + "late_producer_dag.py", # Pure contract/schedule derivation; no treatment. "medicaid_take_up.py", "misc_itemized.py", "nonzero_shares.py", @@ -249,6 +256,10 @@ "take_up.py", "take_up_contract.py", "target_aging.py", + # Data-only final-owner matrix; provenance owner above. + "us_late_overlap_ownership.py", + # Data-only late input/output registry; provenance owner above. + "us_late_producer_registry.py", "validation_input_coverage.py", "warm_start_selection.py", } @@ -3265,8 +3276,8 @@ def test_pool_build_tool_import_graph_is_source_spine_blind() -> None: for tool in _SPINE_BLIND_BUILD_TOOLS: runtime_graph, missing_modules = _us_runtime_import_graph(tool) - assert len(runtime_graph) == 61, ( - f"{tool.name} must reach the pinned 61-module runtime graph; " + assert len(runtime_graph) == 64, ( + f"{tool.name} must reach the pinned 64-module runtime graph; " f"reached {len(runtime_graph)}" ) assert not missing_modules, ( diff --git a/packages/microcosm-build/tests/test_us_stacked_spine.py b/packages/microcosm-build/tests/test_us_stacked_spine.py index 3dd15290..4e1b8f9a 100644 --- a/packages/microcosm-build/tests/test_us_stacked_spine.py +++ b/packages/microcosm-build/tests/test_us_stacked_spine.py @@ -14,6 +14,7 @@ import json import pickle from collections import Counter +from collections.abc import Mapping from copy import deepcopy from dataclasses import FrozenInstanceError, replace from pathlib import Path @@ -24,6 +25,9 @@ from pandas.testing import assert_frame_equal import microcosm.build.us_runtime.acs_income_universe as universe_module +import microcosm.build.us_runtime.acs_transfer as acs_transfer_module +import microcosm.build.us_runtime.multispine_pool as multispine_pool_module +import microcosm.build.us_runtime.puf_capital_gains_tail as tail_module import microcosm.build.us_runtime.puf_support as puf_support_module import microcosm.build.us_runtime.stacked_spine as stacked_spine_module from microcosm.build.frame_checkpoint import ( @@ -35,8 +39,15 @@ from microcosm.build.us_runtime.acs_income_universe import ( apply_acs_pums_earnings_universe_zeros, ) +from microcosm.build.us_runtime.acs_transfer import AcsTransferResult from microcosm.build.us_runtime.acs_transfer_bank import AcsTransferTargetBankStore +from microcosm.build.us_runtime.late_producer_dag import ( + ProducerContract, + ProducerInput, + ProducerInputColumn, +) from microcosm.build.us_runtime.multispine_pool import ( + PoolStageOutput, derive_multispine_pool_inputs, pool_transfer_target_families, ) @@ -84,6 +95,9 @@ support_clone_index_column, support_source_id_column, ) +from microcosm.build.us_runtime.us_late_overlap_ownership import ( + us_late_overlap_ownership_receipt, +) from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights @@ -635,6 +649,157 @@ def _finalize_fixture_predictions( return predictions, donor +def _s_corp_universe_fixture() -> Frame: + cloned = _cloned_stacked_fixture() + person = cloned.table("person").copy(deep=True) + clone_column = support_clone_index_column("person") + clone_index = person[clone_column] + person["s_corp_income"] = np.where(clone_index.eq(0), np.nan, 0.0) + first_clone = person.index[clone_index.eq(1)][0] + person.loc[first_clone, clone_column] = 2 + tables = {entity: cloned.table(entity) for entity in cloned.entities} + tables["person"] = person + return Frame( + tables, + cloned.schema, + {entity: cloned.weights_for(entity) for entity in cloned.weighted_entities}, + cloned.strata, + mass_log=cloned.mass_log, + metadata=cloned.metadata, + ) + + +def test_s_corp_universe_zero_materializes_native_rows_with_exact_receipt() -> None: + frame = _s_corp_universe_fixture() + input_person = frame.table("person").copy(deep=True) + donor = pd.DataFrame({"s_corp_income": [0.0, -0.0, 0.0]}) + + materialized, receipt = ( + stacked_spine_module._materialize_us_puf_s_corp_universe_zero(frame, donor) + ) + + clone_column = support_clone_index_column("person") + clone_index = input_person[clone_column] + native = clone_index.eq(0) + assert input_person.loc[native, "s_corp_income"].isna().all() + assert materialized.table("person")["s_corp_income"].eq(0.0).all() + assert receipt["rule"] == ( + stacked_spine_module.us_puf_s_corp_universe_zero_rule_identity() + ) + assert receipt["status"] == "materialized" + assert receipt["donor_rows_verified"] == 3 + assert receipt["native_rows_materialized"] == int(native.sum()) + assert receipt["produced_rows_verified"] == int((clone_index > 0).sum()) + assert receipt["person_rows"] == len(input_person) + assert receipt["person_rows_by_clone_role"] == { + str(role): int(clone_index.eq(role).sum()) for role in (0, 1, 2) + } + assert receipt["post_materialization_nonfinite_rows"] == 0 + assert receipt["post_materialization_nonzero_rows"] == 0 + assert len(receipt["donor_values_sha256"]) == 64 + assert len(receipt["person_values_sha256"]) == 64 + assert receipt["sha256"] == stacked_spine_module._canonical_sha256( + {key: value for key, value in receipt.items() if key != "sha256"} + ) + + +@pytest.mark.parametrize( + ("mutation", "match"), + ( + ("donor_nonfinite", r"donor precondition failed: 1 nonfinite"), + ("donor_nonzero", r"donor precondition failed: 1 nonzero"), + ("native_preexisting", r"native precondition failed: 1 native cell"), + ("clone_nonfinite", r"clone precondition failed: 1 nonfinite"), + ("tail_nonzero", r"clone precondition failed: 1 nonzero"), + ), +) +def test_s_corp_universe_zero_fails_closed( + mutation: str, + match: str, +) -> None: + frame = _s_corp_universe_fixture() + donor = pd.DataFrame({"s_corp_income": [0.0, 0.0]}) + person = frame.table("person") + clone_index = person[support_clone_index_column("person")] + if mutation == "donor_nonfinite": + donor.loc[0, "s_corp_income"] = np.nan + elif mutation == "donor_nonzero": + donor.loc[0, "s_corp_income"] = 1.0 + elif mutation == "native_preexisting": + person.loc[person.index[clone_index.eq(0)][0], "s_corp_income"] = 0.0 + elif mutation == "clone_nonfinite": + person.loc[person.index[clone_index.eq(1)][0], "s_corp_income"] = np.nan + else: + person.loc[person.index[clone_index.eq(2)][0], "s_corp_income"] = 1.0 + + with pytest.raises(ValueError, match=match): + stacked_spine_module._materialize_us_puf_s_corp_universe_zero(frame, donor) + + +def test_stacked_primary_applies_s_corp_universe_rule_after_qrf( + monkeypatch: pytest.MonkeyPatch, +) -> None: + events: list[str] = [] + + def impute(frame: Frame, *_args: object, **kwargs: object) -> Frame: + events.append("primary_qrf") + receipts = kwargs["predictor_universe_receipts"] + assert isinstance(receipts, list) + receipts.append({"fixture": "recipient-universe"}) + person = frame.table("person").copy(deep=True) + clone_index = person[support_clone_index_column("person")] + person["s_corp_income"] = np.where(clone_index.eq(0), np.nan, 0.0) + tables = {entity: frame.table(entity) for entity in frame.entities} + tables["person"] = person + return Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + + materialize = stacked_spine_module._materialize_us_puf_s_corp_universe_zero + + def materialize_after_qrf( + frame: Frame, + donor: pd.DataFrame, + ) -> tuple[Frame, dict[str, object]]: + events.append("s_corp_universe_zero") + return materialize(frame, donor) + + monkeypatch.setattr( + stacked_spine_module, + "impute_us_puf_tax_detail_support", + impute, + ) + monkeypatch.setattr( + stacked_spine_module, + "_materialize_us_puf_s_corp_universe_zero", + materialize_after_qrf, + ) + + result = stacked_spine_module._run_stacked_puf_pass_without_tail_for_test( + _late_primary_entry(_stacked_gap_fixture()), + pd.DataFrame({"s_corp_income": [0.0, 0.0]}), + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + predictors=(), + person_outputs=("s_corp_income",), + tax_unit_outputs=(), + ) + + assert events == ["primary_qrf", "s_corp_universe_zero"] + assert result.frame.table("person")["s_corp_income"].eq(0.0).all() + assert result.receipt["s_corp_income_universe_zero"]["status"] == "materialized" + assert result.receipt["doctrines"]["whole_pool_output_universes"] == { + "person.s_corp_income": ( + stacked_spine_module.us_puf_s_corp_universe_zero_rule_identity() + ) + } + + def test_finalize_preserve_nulls_keeps_unowned_cells_null() -> None: cloned = _cloned_stacked_fixture() predictions, donor = _finalize_fixture_predictions(cloned) @@ -671,6 +836,81 @@ def test_finalize_preserve_nulls_keeps_unowned_cells_null() -> None: ) +def test_finalize_preserve_nulls_materializes_registry_boolean_outputs() -> None: + cloned = _cloned_stacked_fixture() + registry = stacked_spine_module.CANONICAL_ORIGIN_BATTERY_METRIC_REGISTRY + boolean_outputs = tuple( + column + for column in puf_support_module.PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS + if registry.get(("person", "puf_tax_itemization", column, 0)) + == "boolean_incidence" + ) + assert set(boolean_outputs) == set(US_QBI_BOOLEAN_OUTPUT_COLUMNS) + + tax_unit = cloned.table("tax_unit") + detail_tax_units = tax_unit[support_clone_index_column("tax_unit")].eq(1) + predictions = pd.DataFrame( + { + column: np.ones(int(detail_tax_units.sum()), dtype=np.float64) + for column in boolean_outputs + }, + index=tax_unit.index[detail_tax_units], + ) + donor = pd.DataFrame( + {column: np.asarray([0.0, 1.0], dtype=np.float64) for column in boolean_outputs} + | {"weight": np.ones(2, dtype=np.float64)} + ) + + finalized = finalize_us_puf_tax_detail_predictions( + cloned, + donor, + predictions, + person_outputs=boolean_outputs, + tax_unit_outputs=(), + absent_cells=PUF_ABSENT_CELLS_PRESERVE_NULLS, + ) + + person = finalized.table("person") + detail_people = person[support_clone_index_column("person")].eq(1) + for column in boolean_outputs: + values = person[column] + assert pd.api.types.is_bool_dtype(values.dtype), (column, values.dtype) + assert values.loc[~detail_people].isna().all() + assert values.loc[detail_people].notna().all() + + +def test_finalize_preserve_nulls_rejects_numeric_boolean_materialization() -> None: + cloned = _cloned_stacked_fixture() + column = US_QBI_BOOLEAN_OUTPUT_COLUMNS[0] + person = cloned.table("person") + person[column] = np.zeros(len(person), dtype=np.float64) + tax_unit = cloned.table("tax_unit") + detail_tax_units = tax_unit[support_clone_index_column("tax_unit")].eq(1) + predictions = pd.DataFrame( + {column: np.ones(int(detail_tax_units.sum()), dtype=np.float64)}, + index=tax_unit.index[detail_tax_units], + ) + donor = pd.DataFrame( + {column: np.asarray([0.0, 1.0]), "weight": np.ones(2, dtype=np.float64)} + ) + + with pytest.raises( + TypeError, + match=( + rf"PUF boolean output {column!r} must contain only physical boolean " + r"values.*dtype float64.*builtins\.float" + ), + ): + finalize_us_puf_tax_detail_predictions( + cloned, + donor, + predictions, + person_outputs=(column,), + tax_unit_outputs=(), + absent_cells=PUF_ABSENT_CELLS_PRESERVE_NULLS, + ) + + def test_finalize_legacy_zero_fill_reproduces_the_audited_defect() -> None: """Pin the run-7 boundary: legacy finalization reads absence as zero.""" @@ -844,10 +1084,13 @@ def _cloned_acs_earnings_universe_fixture() -> Frame: "SEMP", ): person.loc[structural, column] = np.nan - return apply_acs_pums_earnings_universe_zeros( - cloned, - boundary="stacked ACS earnings-universe fixture", - ).frame + return stacked_spine_module._materialize_stacked_acs_earnings_universe(cloned).frame + + +def _late_primary_entry(frame: Frame) -> Frame: + """Materialize the declared DAG predecessor for direct primary tests.""" + + return stacked_spine_module._materialize_stacked_acs_earnings_universe(frame).frame def test_strict_recipient_predictors_apply_exact_acs_age_universe() -> None: @@ -1044,6 +1287,7 @@ def test_strict_recipient_predictors_require_raw_acs_universe_authority( def test_puf_finalize_masks_earnings_allocation_to_age_15_plus() -> None: + pytest.importorskip("policyengine_us") frame = _cloned_acs_earnings_universe_fixture() person = frame.table("person") for column in US_QBI_OUTPUT_COLUMNS: @@ -1549,8 +1793,8 @@ def test_canonical_metric_registry_covers_the_declared_131_target_split() -> Non assert len(gap_targets) == 48 assert len(post_puf_targets) == 70 assert len(puf_producer_targets) == 43 - assert len(source_producer_targets) == 30 - assert len(puf_producer_targets & source_producer_targets) == 3 + assert len(source_producer_targets) == 29 + assert len(puf_producer_targets & source_producer_targets) == 2 assert puf_producer_targets | source_producer_targets == post_puf_targets assert gap_targets.isdisjoint(post_puf_targets) assert len(gap_targets | post_puf_targets) == 118 @@ -1562,6 +1806,323 @@ def test_canonical_metric_registry_covers_the_declared_131_target_split() -> Non } & {target for _entity, _family, target, _clone in surface_targets} +def _registry_boolean_targets( + surface: Mapping[str, Mapping[str, tuple[str, ...]]], +) -> set[tuple[str, str]]: + registry = stacked_spine_module.CANONICAL_ORIGIN_BATTERY_METRIC_REGISTRY + return { + (entity, target) + for entity, families in surface.items() + for family, targets in families.items() + for target in targets + if registry[(entity, family, target, 0)] == "boolean_incidence" + } + + +def _transferred_registry_boolean_targets() -> set[tuple[str, str]]: + return _registry_boolean_targets( + stacked_spine_module.CANONICAL_STACKED_GAP_FILL_SURFACE + ) | _registry_boolean_targets( + stacked_spine_module.CANONICAL_STACKED_POST_PUF_TRANSFER_SURFACE + ) + + +def _canonical_registry_checkpoint_frame() -> Frame: + frame = _source_frame( + household_ids=[1, 2, 3], + weights=[1.0, 2.0, 3.0], + stratum="registry_checkpoint", + ) + tables = {entity: frame.table(entity).copy() for entity in frame.entities} + registry = stacked_spine_module.CANONICAL_ORIGIN_BATTERY_METRIC_REGISTRY + nullable_booleans = _transferred_registry_boolean_targets() + string_categories = {"immigration_status_str", "ssn_card_type"} + for (entity, _family, column, _clone_index), metric in sorted(registry.items()): + table = tables[entity] + if metric == "monetary_sign_separated": + values: pd.Series | np.ndarray = np.asarray( + [-1.25, 0.0, 2.5], + dtype=np.float64, + ) + elif metric == "boolean_incidence": + if (entity, column) in nullable_booleans: + values = pd.Series( + [True, pd.NA, False], + index=table.index, + dtype="boolean", + ) + else: + values = np.asarray([True, False, True], dtype=np.bool_) + elif column in string_categories: + values = pd.Series( + ["A", pd.NA, "B"], + index=table.index, + dtype=CANONICAL_STRING_DTYPE, + ) + else: + assert metric == "categorical_tvd" + values = np.asarray([1.0, np.nan, 3.0], dtype=np.float64) + table[column] = values + + person = tables["person"] + for column in ("is_female", "is_household_head"): + person[column] = pd.Series( + [True, False, True], + index=person.index, + dtype="boolean", + ) + return Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + + +def test_canonical_metric_registry_drives_checkpoint_round_trip( + tmp_path: Path, +) -> None: + frame = _canonical_registry_checkpoint_frame() + first_path = tmp_path / "registry-first.h5" + second_path = tmp_path / "registry-second.h5" + + write_frame_checkpoint(first_path, frame) + loaded = load_frame_checkpoint(first_path).frame + write_frame_checkpoint(second_path, loaded) + + assert first_path.read_bytes() == second_path.read_bytes() + registry = stacked_spine_module.CANONICAL_ORIGIN_BATTERY_METRIC_REGISTRY + nullable_booleans = _transferred_registry_boolean_targets() + assert Counter(registry.values()) == { + "monetary_sign_separated": 79, + "boolean_incidence": 48, + "categorical_tvd": 4, + } + for (entity, _family, column, _clone_index), metric in registry.items(): + expected = frame.table(entity)[column] + observed = loaded.table(entity)[column] + pd.testing.assert_series_equal( + observed, + expected, + check_dtype=True, + check_exact=True, + ) + if metric == "boolean_incidence": + if (entity, column) in nullable_booleans: + assert observed.dtype == pd.BooleanDtype() + assert observed.isna().sum() == 1 + else: + assert observed.dtype == np.dtype(np.bool_) + assert not observed.isna().any() + elif metric == "monetary_sign_separated": + assert observed.dtype == np.dtype(np.float64) + elif column in {"immigration_status_str", "ssn_card_type"}: + assert observed.dtype == CANONICAL_STRING_DTYPE + else: + assert observed.dtype == np.dtype(np.float64) + assert observed.isna().sum() == 1 + for column in ("is_female", "is_household_head"): + observed = loaded.person[column] + assert observed.dtype == pd.BooleanDtype() + assert not observed.isna().any() + + +def test_checkpoint_boundary_extension_dtype_inventory_is_exact() -> None: + registry = stacked_spine_module.CANONICAL_ORIGIN_BATTERY_METRIC_REGISTRY + transferred_registry = _transferred_registry_boolean_targets() + source_native = { + ("person", "is_female"), + ("person", "is_household_head"), + } + transferred = transferred_registry | source_native + expected = { + ( + "person", + "attends_eligible_educational_institution_for_american_opportunity_credit", + ), + ("person", "business_is_sstb"), + ("person", "estate_income_would_be_qualified"), + ("person", "farm_operations_income_would_be_qualified"), + ("person", "farm_rent_income_would_be_qualified"), + ("person", "has_american_opportunity_credit_1098_t_or_exception"), + ("person", "has_american_opportunity_credit_institution_ein"), + ("person", "has_champva_health_coverage_at_interview"), + ("person", "has_esi"), + ("person", "has_indian_health_service_coverage_at_interview"), + ("person", "has_marketplace_health_coverage_at_interview"), + ("person", "has_medicaid_health_coverage_at_interview"), + ("person", "has_non_marketplace_direct_purchase_health_coverage_at_interview"), + ("person", "has_other_means_tested_health_coverage_at_interview"), + ("person", "has_tricare_health_coverage_at_interview"), + ("person", "has_va_health_coverage_at_interview"), + ("person", "is_blind"), + ("person", "is_disabled"), + ("person", "is_enrolled_at_least_half_time_for_american_opportunity_credit"), + ("person", "is_female"), + ("person", "is_full_time_college_student"), + ("person", "is_household_head"), + ("person", "is_incapable_of_self_care"), + ("person", "is_pregnant"), + ("person", "is_pursuing_credential_for_american_opportunity_credit"), + ("person", "is_separated"), + ("person", "is_surviving_spouse"), + ("person", "partnership_s_corp_income_would_be_qualified"), + ("person", "previous_year_income_available"), + ("person", "receives_wic"), + ("person", "rental_income_would_be_qualified"), + ("person", "self_employment_income_would_be_qualified"), + ("person", "sstb_self_employment_income_would_be_qualified"), + ("person", "takes_up_medicare_if_eligible"), + ("person", "would_claim_wic"), + ("spm_unit", "is_tanf_enrolled"), + ("spm_unit", "receives_housing_assistance"), + ("spm_unit", "receives_snap"), + ("spm_unit", "takes_up_housing_assistance_if_eligible"), + } + assert len(transferred_registry) == 37 + assert transferred == expected + assert len(transferred) == 39 + + terminal_registry = { + (entity, column) + for (entity, _family, column, _clone_index), metric in registry.items() + if metric == "boolean_incidence" + } + seeded_numpy_booleans = terminal_registry - transferred_registry + assert seeded_numpy_booleans == { + ("person", "takes_up_basic_health_program_if_eligible"), + ("person", "takes_up_chip_if_eligible"), + ("person", "takes_up_early_head_start_if_eligible"), + ("person", "takes_up_head_start_if_eligible"), + ("person", "takes_up_medicaid_if_eligible"), + ("person", "takes_up_ssi_if_eligible"), + ("spm_unit", "takes_up_snap_if_eligible"), + ("spm_unit", "takes_up_tanf_if_eligible"), + ("tax_unit", "takes_up_aca_if_eligible"), + ("tax_unit", "takes_up_dc_ptc"), + ("tax_unit", "takes_up_eitc"), + } + assert len(seeded_numpy_booleans) == 11 + assembled_strings = { + ("person", "PERIDNUM"), + ("person", "source_person_id"), + ("person", "tax_unit_role_input"), + ("person", "person_support_channel"), + ("household", "SERIALNO"), + ("household", "ST"), + ("household", "PUMA"), + ("household", "puma_geoid"), + ("household", "puma"), + ("household", "tenure_type"), + ("household", "household_support_channel"), + ("tax_unit", "filing_status_input"), + ("tax_unit", "tax_unit_support_channel"), + ("spm_unit", "spm_unit_tenure_type"), + ("spm_unit", "spm_unit_support_channel"), + ("family", "family_support_channel"), + ("marital_unit", "marital_unit_support_channel"), + } + transferred_strings = assembled_strings | { + ("person", "immigration_status_str"), + ("person", "ssn_card_type"), + } + boundary_inventory = { + "assembled": { + "boolean": frozenset(), + "string": frozenset(assembled_strings), + }, + "transferred": { + "boolean": frozenset(transferred), + "string": frozenset(transferred_strings), + }, + "simulated": { + "boolean": frozenset(transferred), + "string": frozenset(transferred_strings), + }, + } + assert { + stage: {family: len(columns) for family, columns in families.items()} + for stage, families in boundary_inventory.items() + } == { + "assembled": {"boolean": 0, "string": 17}, + "transferred": {"boolean": 39, "string": 19}, + "simulated": {"boolean": 39, "string": 19}, + } + assert sum(map(len, boundary_inventory["assembled"].values())) == 17 + assert sum(map(len, boundary_inventory["transferred"].values())) == 58 + assert boundary_inventory["transferred"] == boundary_inventory["simulated"] + + +def test_registry_drives_every_late_callback_dtype_family_check() -> None: + registry = stacked_spine_module.CANONICAL_ORIGIN_BATTERY_METRIC_REGISTRY + by_column = { + (entity, column): metric + for (entity, _family, column, _clone_index), metric in registry.items() + } + assert len(by_column) == len(registry) == 131 + + representative = { + "monetary_sign_separated": pd.Series([1.0, pd.NA], dtype="Float64"), + "boolean_incidence": pd.Series([True, pd.NA], dtype="boolean"), + "categorical_tvd": pd.Series([1, pd.NA], dtype="Int64"), + } + wrong = { + "monetary_sign_separated": pd.Series([True], dtype=bool), + "boolean_incidence": pd.Series([1.0], dtype=np.float64), + "categorical_tvd": pd.Series([True], dtype=bool), + } + for metric in registry.values(): + assert stacked_spine_module._late_output_matches_metric_family( + representative[metric], + metric, + ) + assert not stacked_spine_module._late_output_matches_metric_family( + wrong[metric], + metric, + ) + assert stacked_spine_module._late_output_matches_metric_family( + pd.Series(["NON_CITIZEN", pd.NA], dtype="string"), + "categorical_tvd", + ) + assert stacked_spine_module._late_output_matches_metric_family( + pd.Series(pd.Categorical(["A", "B"])), + "categorical_tvd", + ) + assert not stacked_spine_module._late_output_matches_metric_family( + pd.Series([True], dtype=object), + "boolean_incidence", + ) + + registered_occurrences = [ + (contract.name, output.entity, output.column, by_column[key]) + for contract in ( + stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY.values() + ) + for output in contract.outputs + if (key := (output.entity, output.column)) in by_column + ] + assert len(registered_occurrences) == 163 + assert Counter( + metric for _producer, _entity, _column, metric in registered_occurrences + ) == { + "monetary_sign_separated": 120, + "boolean_incidence": 37, + "categorical_tvd": 6, + } + unique_late_targets = { + (entity, column): metric + for _producer, entity, column, metric in registered_occurrences + } + assert len(unique_late_targets) == 90 + assert Counter(unique_late_targets.values()) == { + "monetary_sign_separated": 67, + "boolean_incidence": 20, + "categorical_tvd": 3, + } + + def test_explicit_test_seams_reject_the_canonical_authority() -> None: authority = stacked_spine_module._production_stacked_authority() frame = _stacked_gap_fixture() @@ -2322,154 +2883,2467 @@ def test_donor_byte_identity_canonicalizes_semantic_strings() -> None: boundary=f"{control.name} donor identity after", ) - mixed = pd.Series( - ["RENTED", 1], - name="mixed_tenure_type", - dtype=object, + mixed = pd.Series( + ["RENTED", 1], + name="mixed_tenure_type", + dtype=object, + ) + with pytest.raises(TypeError, match="semantic strings cannot mix"): + stacked_spine_module._canonical_donor_series_payload( + mixed, + boundary="mixed-object donor identity", + ) + + +def test_donor_byte_identity_ignores_string_object_alias_topology() -> None: + shared = "".join(["dynamically", "-", "allocated", "-", "donor"]) + separate = [ + "".join(["dynamically", "-", "allocated", "-", "donor"]) for _ in range(2) + ] + before = pd.Series([shared, shared], name="native_label", dtype=object) + after = pd.Series(separate, name="native_label", dtype=object) + assert before.iloc[0] is before.iloc[1] + assert after.iloc[0] is not after.iloc[1] + + assert stacked_spine_module._canonical_donor_series_payload( + before, + boundary="aliased-string donor identity before", + ) == stacked_spine_module._canonical_donor_series_payload( + after, + boundary="aliased-string donor identity after", + ) + + +def test_donor_byte_identity_accepts_semantic_boolean_object_scalars() -> None: + semantic_booleans = pd.Series( + [True, np.bool_(False), None], + name="is_disabled", + dtype=object, + ) + + assert stacked_spine_module._canonical_donor_series_payload( + semantic_booleans, + boundary="semantic-boolean donor identity before", + ) == stacked_spine_module._canonical_donor_series_payload( + semantic_booleans.copy(deep=True), + boundary="semantic-boolean donor identity after", + ) + + +def test_gap_fill_activation_authority_fails_closed_on_donor_nulls() -> None: + stacked = _stacked_gap_fixture() + person = stacked.table("person").copy() + channel = person[support_channel_column("person")] + poke = person.index[channel.eq("asec")][1] + person.loc[poke, "unemployment_compensation"] = np.nan + tables = {entity: stacked.table(entity) for entity in stacked.entities} + tables["person"] = person + poked = Frame( + tables, + stacked.schema, + {entity: stacked.weights_for(entity) for entity in stacked.weighted_entities}, + stacked.strata, + mass_log=stacked.mass_log, + metadata=stacked.metadata, + ) + + with pytest.raises(ValueError, match="donors must observe"): + _gap_fill_with_test_authority(poked, plan=_GAP_FILL_TEST_PLAN, seed=578) + + +@pytest.mark.parametrize( + ("recipient_channel", "donor_channel", "role", "missing_channel"), + ( + ("acx_typo", "asec", "recipient", "acx_typo"), + ("acs", "asec_typo", "donor", "asec_typo"), + ), +) +def test_gap_fill_activation_authority_rejects_nonlive_declared_channel( + recipient_channel: str, + donor_channel: str, + role: str, + missing_channel: str, +) -> None: + plan = ( + GapFillDirection( + name="nonlive_channel", + recipient_channel=recipient_channel, + donor_channel=donor_channel, + target_families={"person": {"complete": ("age",)}}, + ), + ) + + with pytest.raises( + ValueError, + match=f"declared {role} channel {missing_channel!r} has no live rows", + ): + _gap_fill_with_test_authority(_stacked_gap_fixture(), plan=plan, seed=578) + + +def test_gap_fill_fails_closed_on_missing_target_column() -> None: + stacked = _stacked_gap_fixture() + plan = ( + GapFillDirection( + name="asec_survey_to_acs", + recipient_channel="acs", + donor_channel="asec", + target_families={ + "person": {"model_required_numeric": ("veterans_benefits",)} + }, + ), + ) + with pytest.raises(ValueError, match="veterans_benefits.*absent"): + _gap_fill_with_test_authority(stacked, plan=plan, seed=578) + + +def test_gap_fill_rejects_cloned_frames() -> None: + cloned = clone_us_frame_for_puf_support(_stacked_gap_fixture()) + with pytest.raises(ValueError, match="before clone operators"): + _gap_fill_with_test_authority(cloned, plan=_GAP_FILL_TEST_PLAN, seed=578) + + +def _post_puf_transfer_fixture() -> Frame: + attached = clone_us_frame_for_puf_support( + _stacked_gap_fixture(), + clone_attachment_fraction=0.75, + clone_attachment_seed=578, + ) + person = attached.table("person").copy() + source_producer_rows = ( + person[support_channel_column("person")].astype(str).eq("asec") + ) + person["is_pregnant"] = pd.Series( + pd.NA, + index=person.index, + dtype="boolean", + ) + person.loc[source_producer_rows, "is_pregnant"] = np.resize( + np.asarray([True, False]), + int(source_producer_rows.sum()), + ) + tables = {entity: attached.table(entity) for entity in attached.entities} + tables["person"] = person + return Frame( + tables, + attached.schema, + {entity: attached.weights_for(entity) for entity in attached.weighted_entities}, + attached.strata, + mass_log=attached.mass_log, + metadata=attached.metadata, + ) + + +def test_bounded_transfer_group_remaps_canonical_producer_roles() -> None: + group = next( + group + for group in stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS + if group.family == "puf_tax_itemization__batch_2" + ) + + puf_roles = stacked_spine_module._producer_role_surface_for_group( + group.target_families, + stacked_spine_module.CANONICAL_STACKED_POST_PUF_PUF_PRODUCER_SURFACE, + ) + source_roles = stacked_spine_module._producer_role_surface_for_group( + group.target_families, + stacked_spine_module.CANONICAL_STACKED_POST_PUF_SOURCE_PRODUCER_SURFACE, + ) + + assert "qualified_tuition_expenses" in puf_roles["person"][group.family] + assert "traditional_ira_contributions_desired" in puf_roles["person"][group.family] + assert source_roles == { + "person": { + group.family: ("traditional_ira_contributions_desired",), + } + } + + +def test_late_readiness_rejects_object_typed_nonfinite_numeric_input() -> None: + frame = _post_puf_transfer_fixture() + person = frame.table("person").copy() + person["late_numeric"] = pd.Series( + np.resize(np.asarray([np.inf, "bad"], dtype=object), len(person)), + index=person.index, + dtype=object, + ) + tables = {entity: frame.table(entity) for entity in frame.entities} + tables["person"] = person + poisoned = Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + requirement = ProducerInput( + "person", + "late_numeric", + "asec_source", + "transfer:fixture", + alternatives=( + ( + ProducerInputColumn( + "person", + "late_numeric", + "finite_numeric", + ), + ), + ), + ) + contract = ProducerContract( + "source:fixture", + "post_clone_source", + (requirement,), + (), + ) + + unfilled, invalid = stacked_spine_module._late_input_readiness_rows( + poisoned, + contract, + ) + + assert unfilled[requirement] == 0 + assert invalid[requirement] == int( + person[support_channel_column("person")].astype(str).eq("asec").sum() + ) + with pytest.raises( + ValueError, + match=r"person\.late_numeric.*transfer:fixture", + ): + stacked_spine_module.run_producer_when_ready( + contract, + lambda: pytest.fail("invalid numeric input reached callback"), + unfilled_rows=unfilled, + invalid_rows=invalid, + absence_receipts={}, + ) + + +def _typehugq_cross_origin_readiness_fixture() -> tuple[ + Frame, + ProducerContract, + ProducerInput, +]: + asec_household_count = 1_688 + asec = _source_frame( + household_ids=list(range(1, asec_household_count + 1)), + weights=[1.0] * asec_household_count, + extra_person_columns={"asec_detail_income": 40.0}, + stratum="asec_2024", + ) + frame = assemble_stacked_spine( + asec, + _acs_source(), + acs_sample_fraction=1.0, + acs_sample_seed=578, + ).frame + primary = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY[ + stacked_spine_module.US_LATE_PRIMARY_PUF_STAGE + ] + requirement = next( + item + for item in primary.inputs + if item.column == "@effective:validated_structure:TYPEHUGQ" + ) + contract = replace(primary, inputs=(requirement,), outputs=()) + return frame, contract, requirement + + +def test_typehugq_accepts_exact_1688_asec_structural_null_rows() -> None: + frame, contract, requirement = _typehugq_cross_origin_readiness_fixture() + household = frame.table("household") + support_channel = household[support_channel_column("household")].astype(str) + asec_rows = support_channel.eq("asec") + acs_rows = support_channel.eq("acs") + + assert int(asec_rows.sum()) == 1_688 + assert int(acs_rows.sum()) == 10 + assert int(household.loc[asec_rows, "TYPEHUGQ"].isna().sum()) == 1_688 + assert int(household.loc[acs_rows, "TYPEHUGQ"].isna().sum()) == 0 + assert requirement.required_scope == "acs_source" + assert requirement.tolerated_absence_receipts == () + + unfilled, invalid = stacked_spine_module._late_input_readiness_rows( + frame, + contract, + ) + absence = stacked_spine_module._late_declared_absence_receipts( + contract, + unfilled, + invalid_rows=invalid, + ) + + assert unfilled == {requirement: 0} + assert invalid == {requirement: 0} + assert absence == {} + assert ( + stacked_spine_module.run_producer_when_ready( + contract, + lambda: "ran", + unfilled_rows=unfilled, + invalid_rows=invalid, + absence_receipts=absence, + ) + == "ran" + ) + + +def test_typehugq_still_refuses_one_missing_acs_row() -> None: + frame, contract, requirement = _typehugq_cross_origin_readiness_fixture() + household = frame.table("household").copy() + support_channel = household[support_channel_column("household")].astype(str) + acs_row = household.index[support_channel.eq("acs")][0] + household.loc[acs_row, "TYPEHUGQ"] = np.nan + tables = {entity: frame.table(entity) for entity in frame.entities} + tables["household"] = household + missing = Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + + unfilled, invalid = stacked_spine_module._late_input_readiness_rows( + missing, + contract, + ) + absence = stacked_spine_module._late_declared_absence_receipts( + contract, + unfilled, + invalid_rows=invalid, + ) + invoked = False + + def callback() -> None: + nonlocal invoked + invoked = True + + assert unfilled == {requirement: 1} + assert invalid == {requirement: 0} + assert absence == {} + with pytest.raises( + ValueError, + match=( + r"(?s)primary_puf_qrf.*" + r"household\.@effective:validated_structure:TYPEHUGQ.*" + r"1 unfilled.*acs_source.*post_clone_input_surface.*" + r"tolerated absence receipts=\[\]" + ), + ): + stacked_spine_module.run_producer_when_ready( + contract, + callback, + unfilled_rows=unfilled, + invalid_rows=invalid, + absence_receipts=absence, + ) + assert invoked is False + + +def _fill_late_contract_surface( + frame: Frame, + *, + contracts: tuple[ProducerContract, ...], + include_outputs: bool, +) -> Frame: + tables = {entity: frame.table(entity).copy() for entity in frame.entities} + metric_by_column = { + (entity, column): metric + for ( + entity, + _family, + column, + _clone_index, + ), metric in stacked_spine_module.CANONICAL_ORIGIN_BATTERY_METRIC_REGISTRY.items() + } + owners = { + column: entity for entity, table in tables.items() for column in table.columns + } + for contract in contracts: + columns: list[ProducerInputColumn] = [] + for requirement in contract.inputs: + selected: tuple[ProducerInputColumn, ...] = () + for alternative in requirement.alternatives: + physical = tuple( + column + for column in alternative + if not column.column.startswith("@") + ) + if not physical and any( + column.column != "@resolved_weight" for column in alternative + ): + continue + if all( + column.column not in owners + or owners[column.column] == column.entity + for column in physical + ): + selected = physical + break + columns.extend(selected) + owners.update((column.column, column.entity) for column in selected) + if include_outputs: + for output in contract.outputs: + if output.entity == "frame" or output.column.startswith("@"): + continue + columns.append(ProducerInputColumn(output.entity, output.column)) + owners[output.column] = output.entity + for column in columns: + table = tables[column.entity] + if ( + metric_by_column.get((column.entity, column.column)) + == "boolean_incidence" + ): + table[column.column] = pd.Series( + True, + index=table.index, + dtype="boolean", + ) + elif column.column in table: + table[column.column] = table[column.column].fillna(1) + elif column.column != "person_support_clone_index": + table[column.column] = 1.0 + return Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + + +def test_canonical_transfer_rejects_nonfinite_optional_numeric_as_invalid() -> None: + contract = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY[ + "transfer:person/adult_care" + ] + complete = _fill_late_contract_surface( + _post_puf_transfer_fixture(), + contracts=(contract,), + include_outputs=False, + ) + person = complete.table("person").copy() + person.loc[person.index[0], "employment_income_before_lsr"] = np.inf + tables = {entity: complete.table(entity) for entity in complete.entities} + tables["person"] = person + poisoned = Frame( + tables, + complete.schema, + {entity: complete.weights_for(entity) for entity in complete.weighted_entities}, + complete.strata, + mass_log=complete.mass_log, + metadata=complete.metadata, + ) + requirement = next( + item + for item in contract.inputs + if item.column == "@effective:optional_employment_income" + ) + + unfilled, invalid = stacked_spine_module._late_input_readiness_rows( + poisoned, + contract, + ) + absence = stacked_spine_module._late_declared_absence_receipts( + contract, + unfilled, + invalid_rows=invalid, + ) + + assert unfilled[requirement] == 0 + assert invalid[requirement] == 1 + assert requirement.tolerated_absence_receipts[0] not in absence + with pytest.raises( + ValueError, + match=( + r"(?s)transfer:person/adult_care.*" + r"person\.@effective:optional_employment_income.*1 invalid.*" + r"post_clone_input_surface" + ), + ): + stacked_spine_module.run_producer_when_ready( + contract, + lambda: pytest.fail("invalid predictor reached transfer callback"), + unfilled_rows=unfilled, + invalid_rows=invalid, + absence_receipts=absence, + ) + + +def test_person_transfer_refuses_invalid_peer_grain_provenance_before_fit() -> None: + contract = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY[ + "transfer:person/adult_care" + ] + complete = _fill_late_contract_surface( + _post_puf_transfer_fixture(), + contracts=(contract,), + include_outputs=False, + ) + family = complete.table("family").copy() + family["family_support_clone_index"] = family["family_support_clone_index"].astype( + float + ) + family.loc[family.index[0], "family_support_clone_index"] = np.inf + tables = {entity: complete.table(entity) for entity in complete.entities} + tables["family"] = family + poisoned = Frame( + tables, + complete.schema, + {entity: complete.weights_for(entity) for entity in complete.weighted_entities}, + complete.strata, + mass_log=complete.mass_log, + metadata=complete.metadata, + ) + direct = next( + item + for item in contract.inputs + if item.entity == "family" + and item.column == "family_support_clone_index" + and item.producing_stage == stacked_spine_module.US_LATE_PRIMARY_PUF_STAGE + ) + + unfilled, invalid = stacked_spine_module._late_input_readiness_rows( + poisoned, + contract, + ) + + assert unfilled[direct] == 0 + assert invalid[direct] == 1 + with pytest.raises( + ValueError, + match=( + r"(?s)transfer:person/adult_care.*" + r"family\.family_support_clone_index.*1 invalid.*primary_puf_qrf" + ), + ): + stacked_spine_module.run_producer_when_ready( + contract, + lambda: pytest.fail("peer-grain poison reached transfer fit"), + unfilled_rows=unfilled, + invalid_rows=invalid, + absence_receipts={}, + ) + + +@pytest.mark.parametrize( + ("metadata_key", "producing_stage"), + ( + ("us_spine_assembly_manifest", "post_clone_input_surface"), + ("us_stacked_spine_manifest", "post_clone_input_surface"), + ("us_puf_clone_attachment_manifest", "primary_puf_qrf"), + ), +) +def test_transfer_refuses_missing_validation_metadata_before_fit( + metadata_key: str, + producing_stage: str, +) -> None: + contract = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY[ + "transfer:person/adult_care" + ] + complete = _fill_late_contract_surface( + _post_puf_transfer_fixture(), + contracts=(contract,), + include_outputs=False, + ) + missing = Frame( + {entity: complete.table(entity) for entity in complete.entities}, + complete.schema, + {entity: complete.weights_for(entity) for entity in complete.weighted_entities}, + complete.strata, + mass_log=complete.mass_log, + metadata={ + key: value + for key, value in complete.metadata.items() + if key != metadata_key + }, + ) + requirement = next( + item + for item in contract.inputs + if item.producing_stage == producing_stage + and any( + column.entity == "frame" and column.column == f"@{metadata_key}" + for alternative in item.alternatives + for column in alternative + ) + ) + + unfilled, invalid = stacked_spine_module._late_input_readiness_rows( + missing, + contract, + ) + + assert unfilled[requirement] == 1 + assert invalid[requirement] == 0 + with pytest.raises( + ValueError, + match=rf"(?s)transfer:person/adult_care.*{producing_stage}", + ): + stacked_spine_module.run_producer_when_ready( + contract, + lambda: pytest.fail("missing metadata reached transfer fit"), + unfilled_rows=unfilled, + invalid_rows=invalid, + absence_receipts={}, + ) + + +def _late_table_digest_vector() -> pd.DataFrame: + return pd.DataFrame( + { + "boolean": pd.array([True, False, pd.NA], dtype="boolean"), + "integer": pd.array([1, -2, pd.NA], dtype="Int64"), + "float": np.array([np.inf, -0.0, np.nan], dtype=np.float64), + "string": pd.array(["", "café", pd.NA], dtype=CANONICAL_STRING_DTYPE), + }, + index=pd.Index([7, 3, 11], dtype=np.int64, name="row_id"), + ) + + +def test_late_table_content_digest_is_byte_stable_for_typed_scalar_vector() -> None: + table = _late_table_digest_vector() + + first = stacked_spine_module._late_table_values_sha256(table) + second = stacked_spine_module._late_table_values_sha256(table.copy(deep=True)) + + assert ( + first + == second + == ("da35b24dd68ac7a8917e27c37c81a44d8ab4fbc4888539e29999f904ce741254") + ) + assert len(first) == 64 + assert set(first) <= set("0123456789abcdef") + + +@pytest.mark.parametrize( + ("left", "right"), + ( + ([True], [1]), + ([1], [1.0]), + ([None], [""]), + (["ab", "c"], ["a", "bc"]), + ([0.0], [-0.0]), + ), +) +def test_late_table_content_digest_domain_separates_object_scalars( + left: list[object], + right: list[object], +) -> None: + left_table = pd.DataFrame({"value": pd.Series(left, dtype=object)}) + right_table = pd.DataFrame({"value": pd.Series(right, dtype=object)}) + + assert stacked_spine_module._late_table_values_sha256( + left_table + ) != stacked_spine_module._late_table_values_sha256(right_table) + + +def test_late_table_content_digest_canonicalizes_null_float_payloads() -> None: + ordinary_nan = np.array([np.nan], dtype=np.float64) + alternate_nan = np.array([0x7FF8_0000_0000_0001], dtype=" None: + index = pd.Index([9, 4], dtype=np.int64, name="row_id") + object_strings = pd.DataFrame( + { + "label": pd.Series( + ["RENTED", None], + index=index, + dtype=object, + ) + }, + index=index, + ) + canonical_strings = pd.DataFrame( + { + "label": pd.Series( + ["RENTED", pd.NA], + index=index, + dtype=CANONICAL_STRING_DTYPE, + ) + }, + index=index, + ) + + assert stacked_spine_module._late_table_values_sha256( + object_strings, + normalize_strings=True, + ) == stacked_spine_module._late_table_values_sha256( + canonical_strings, + normalize_strings=True, + ) + assert stacked_spine_module._late_table_values_sha256( + object_strings, + normalize_strings=False, + ) != stacked_spine_module._late_table_values_sha256( + canonical_strings, + normalize_strings=False, + ) + + +def test_late_table_content_digest_binds_dtype_index_and_order() -> None: + base = pd.DataFrame( + {"value": np.array([1, 2], dtype=np.int32)}, + index=pd.Index([4, 8], name="row_id"), + ) + digest = stacked_spine_module._late_table_values_sha256(base) + variants = ( + base.astype({"value": np.int64}), + base.iloc[::-1], + base.rename_axis("different_index"), + base.rename(columns={"value": "different_column"}), + ) + + assert all( + stacked_spine_module._late_table_values_sha256(variant) != digest + for variant in variants + ) + + +def test_late_primary_resources_bind_donor_content_and_execution_config( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.delenv("POPULACE_FIT_N_JOBS", raising=False) + monkeypatch.setenv("POPULACE_FIT_PREDICT_WORKERS", "2") + donor = pd.DataFrame( + {"income": np.array([10.0, 20.0], dtype=np.float64)}, + index=pd.Index([3, 9], name="donor_id"), + ) + common = { + "primary_qrf_checkpoint_identity_sha256": "a" * 64, + "clone_attachment_fraction": 1.0, + "clone_attachment_seed": 578, + "seed": 0, + "n_estimators": 100, + "fit_records_enabled": True, + "tail_bound_diagnostics_enabled": True, + } + + baseline = stacked_spine_module.stacked_late_primary_resource_receipts( + donor, + **common, + ) + changed_donor = donor.copy() + changed_donor.iloc[0, 0] = 11.0 + donor_variant = stacked_spine_module.stacked_late_primary_resource_receipts( + changed_donor, + **common, + ) + config_variant = stacked_spine_module.stacked_late_primary_resource_receipts( + donor, + **{**common, "clone_attachment_seed": 579}, + ) + monkeypatch.setenv("POPULACE_FIT_PREDICT_WORKERS", "3") + environment_variant = stacked_spine_module.stacked_late_primary_resource_receipts( + donor, + **common, + ) + monkeypatch.setenv("POPULACE_FIT_PREDICT_WORKERS", "2") + monkeypatch.setenv("FIXTURE_UNRELATED_SECRET", "must-not-enter-identity") + unrelated_environment_variant = ( + stacked_spine_module.stacked_late_primary_resource_receipts( + donor, + **common, + ) + ) + + assert set(baseline) == { + "tax_unit.@puf_donor_tax_units", + "tax_unit.@primary_qrf_checkpoint", + "tax_unit.@primary_puf_execution_config", + } + donor_receipt = baseline["tax_unit.@puf_donor_tax_units"] + assert ( + donor_receipt["binding"]["table_content_sha256"] + != ( + donor_variant["tax_unit.@puf_donor_tax_units"]["binding"][ + "table_content_sha256" + ] + ) + ) + assert ( + donor_receipt["rows"] == donor_variant["tax_unit.@puf_donor_tax_units"]["rows"] + ) + assert ( + baseline["tax_unit.@primary_puf_execution_config"]["binding_sha256"] + != config_variant["tax_unit.@primary_puf_execution_config"]["binding_sha256"] + ) + qrf = baseline["tax_unit.@primary_puf_execution_config"]["binding"]["qrf"] + assert qrf["predictors"] == list( + stacked_spine_module.PUF_TAX_DETAIL_DEFAULT_PREDICTORS + ) + assert qrf["person_outputs"] == list( + stacked_spine_module.PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS + ) + assert qrf["tax_unit_outputs"] == list( + stacked_spine_module.PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS + ) + assert qrf["invocation_mode"] == { + "predictors": "canonical_default", + "person_outputs": "canonical_default", + "tax_unit_outputs": "canonical_default", + } + execution = baseline["tax_unit.@primary_puf_execution_config"]["binding"] + assert execution["schema_version"] == 4 + assert execution["clone_attachment"]["support_channels"] == [ + stacked_spine_module.BASE_ASEC_SUPPORT_CHANNEL, + stacked_spine_module.PUF_TAX_DETAIL_SUPPORT_CHANNEL, + ] + assert execution["capital_gains_tail"]["spec"] == ( + stacked_spine_module.puf_capital_gains_tail_spec_identity() + ) + assert execution["qrf"]["tail_bound_quantiles"] == { + "non_sch_d_capital_gains": 0.999 + } + assert execution["doctrines"]["whole_pool_output_universes"] == { + "person.s_corp_income": { + "rule_id": "puf_tax_detail_s_corp_income_universe_zero_v1", + "schema_version": 1, + "entity": "person", + "column": "s_corp_income", + "coverage_scope": "whole_pool", + "materialized_value": 0.0, + "source_semantics": ( + "puf_combined_partnership_s_corp_carried_by_partnership_income" + ), + "donor_precondition": "finite_exact_zero", + "puf_clone_precondition": "finite_exact_zero", + "native_precondition": "all_null", + "assignment": "explicit_array_assignment", + } + } + worker = execution["qrf"]["worker_execution"] + assert worker["module"] == "microcosm.build.us_runtime.puf_qrf_worker" + assert worker["argv_template"] == [ + worker["interpreter"]["executable"], + "-m", + "microcosm.build.us_runtime.puf_qrf_worker", + "--checkpoint-dir", + "{checkpoint_dir}", + "--target-index", + "{target_index}", + ] + environment = worker["environment"] + assert environment["policy"] == ( + "inherit_parent_environment_with_bound_fit_controls" + ) + assert environment["overrides"] == {} + assert environment["bound_names"] == [ + "POPULACE_FIT_N_JOBS", + "POPULACE_FIT_PREDICT_WORKERS", + ] + assert environment["semantic_controls"] == { + "POPULACE_FIT_N_JOBS": {"configured": None, "resolved": -1}, + "POPULACE_FIT_PREDICT_WORKERS": { + "configured": "2", + "resolved": 2, + "resolution": "environment_override", + }, + } + assert execution["capital_gains_tail"]["soi_e19200_agi_bands"]["asset_sha256"] + assert execution["capital_gains_tail"]["concentration_gate"] == { + "schema_version": 2, + "selection_quantile": 0.995, + "selection_comparison": "strictly_greater_than", + "reference_quantile": 0.999, + "recipient_capital_gains_topcode": 1_999_998.0, + "positive_mass_five_x_target": 1_270_900_000_000.0, + "worsening_share_tolerance": 1e-9, + "ordered_recipient_agi_proxy_columns": [ + "employment_income_before_lsr", + "self_employment_income_before_lsr", + "taxable_interest_income", + "qualified_dividend_income", + "non_qualified_dividend_income", + "short_term_capital_gains", + "long_term_capital_gains_before_response", + ], + "ordered_joint_vector_columns": [ + "short_term_capital_gains", + "long_term_capital_gains_before_response", + "long_term_capital_gains_on_collectibles", + "non_sch_d_capital_gains", + "unrecaptured_section_1250_gain", + ], + "recipient_owned_candidate_overlap": sorted( + set(stacked_spine_module.PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS) + - set(PUF_CAPITAL_GAINS_TAIL_TAX_UNIT_COLUMNS) + ), + "top_k": 100, + "max_top_share": 0.75, + "min_nonzero_records": 500, + "reviewed_exclusions": {}, + } + assert execution["audit_sinks"] == { + "fit_records": "enabled", + "tail_bound_diagnostics": "enabled", + "recipient_predictor_universe": "required_receipt", + } + assert ( + baseline["tax_unit.@primary_puf_execution_config"]["binding_sha256"] + != environment_variant["tax_unit.@primary_puf_execution_config"][ + "binding_sha256" + ] + ) + assert ( + baseline["tax_unit.@primary_puf_execution_config"]["binding_sha256"] + == unrelated_environment_variant["tax_unit.@primary_puf_execution_config"][ + "binding_sha256" + ] + ) + + +@pytest.mark.parametrize( + ("name", "replacement"), + ( + ("PUF_CAPITAL_GAINS_TAIL_QUANTILE", 0.994), + ("PUF_CAPITAL_GAINS_TAIL_REFERENCE_QUANTILE", 0.998), + ("PUF_CAPITAL_GAINS_TAIL_ASEC_CAPITAL_GAINS_TOPCODE", 2_000_000.0), + ("PUF_CAPITAL_GAINS_TAIL_POSITIVE_MASS_FIVE_X_TARGET", 1.0), + ("PUF_CAPITAL_GAINS_TAIL_WORSENING_SHARE_TOLERANCE", 2e-9), + ("PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_TOP_K", 101), + ("PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_MAX_TOP_SHARE", 0.74), + ("PUF_CAPITAL_GAINS_TAIL_CONCENTRATION_MIN_NONZERO_RECORDS", 501), + ( + "_RECIPIENT_AGI_PROXY_COLUMNS", + tuple(reversed(tail_module._RECIPIENT_AGI_PROXY_COLUMNS)), + ), + ( + "_JOINT_VECTOR_COLUMNS", + tuple(reversed(tail_module._JOINT_VECTOR_COLUMNS)), + ), + ("_RECIPIENT_OWNED_CANDIDATE_OVERLAP", frozenset()), + ), +) +def test_late_primary_resource_identity_binds_every_tail_control( + monkeypatch: pytest.MonkeyPatch, + name: str, + replacement: object, +) -> None: + donor = pd.DataFrame({"fixture_donor": [1.0]}) + common = { + "primary_qrf_checkpoint_identity_sha256": "a" * 64, + "clone_attachment_fraction": 1.0, + "clone_attachment_seed": 578, + "seed": 0, + "n_estimators": 100, + "fit_records_enabled": True, + "tail_bound_diagnostics_enabled": True, + } + baseline = stacked_spine_module.stacked_late_primary_resource_receipts( + donor, + **common, + ) + monkeypatch.setattr(tail_module, name, replacement) + changed = stacked_spine_module.stacked_late_primary_resource_receipts( + donor, + **common, + ) + + assert ( + baseline["tax_unit.@primary_puf_execution_config"]["binding_sha256"] + != changed["tax_unit.@primary_puf_execution_config"]["binding_sha256"] + ) + + +def test_stacked_primary_reuses_one_resolved_tail_input_snapshot( + monkeypatch: pytest.MonkeyPatch, +) -> None: + spec, agi_bands = tail_module.resolve_puf_capital_gains_tail_execution_inputs() + monkeypatch.setattr( + stacked_spine_module, + "resolve_puf_capital_gains_tail_execution_inputs", + lambda: (spec, agi_bands), + ) + + def impute(frame: Frame, *_args: object, **kwargs: object) -> Frame: + receipts = kwargs["predictor_universe_receipts"] + assert isinstance(receipts, list) + receipts.append({"fixture": "recipient-universe"}) + return frame + + observed: dict[str, object] = {} + + def transfer( + _frame: Frame, + _donor: pd.DataFrame, + **kwargs: object, + ) -> tuple[Frame, dict[str, object]]: + observed.update(kwargs) + raise RuntimeError("tail snapshot observed") + + monkeypatch.setattr( + stacked_spine_module, "impute_us_puf_tax_detail_support", impute + ) + monkeypatch.setattr( + stacked_spine_module, "transfer_puf_capital_gains_tail", transfer + ) + + with pytest.raises(RuntimeError, match="tail snapshot observed"): + stacked_spine_module.run_stacked_puf_pass( + _late_primary_entry(_stacked_gap_fixture()), + pd.DataFrame({"fixture_donor": [1.0]}), + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + ) + + assert observed["spec"] is spec + assert observed["agi_bands"] is agi_bands + + +def test_stacked_primary_qrf_refuses_unbound_surface_and_missing_audit_sink() -> None: + donor = pd.DataFrame({"fixture_donor": [1.0]}) + resources = stacked_spine_module.stacked_late_primary_resource_receipts( + donor, + primary_qrf_checkpoint_identity_sha256="a" * 64, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + seed=0, + n_estimators=100, + fit_records_enabled=True, + tail_bound_diagnostics_enabled=True, + ) + binding = stacked_spine_module.stacked_late_primary_checkpoint_input_binding( + resources + ) + frame = _late_primary_entry(_stacked_gap_fixture()) + + with pytest.raises(ValueError, match=r"canonical predictor/output surface"): + stacked_spine_module._run_stacked_puf_pass_without_tail_for_test( + frame, + donor, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + person_outputs=("taxable_interest_income",), + fit_records=[], + tail_bound_diagnostics=[], + primary_qrf_checkpoint_dir=Path("fixture-primary-qrf"), + primary_qrf_input_binding=binding, + ) + + with pytest.raises( + ValueError, + match=r"declared audit sink\(s\).*fit_records", + ): + stacked_spine_module._run_stacked_puf_pass_without_tail_for_test( + frame, + donor, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + tail_bound_diagnostics=[], + primary_qrf_checkpoint_dir=Path("fixture-primary-qrf"), + primary_qrf_input_binding=binding, + ) + + +def test_late_primary_resource_rejects_shallow_receipt_before_callback() -> None: + contract = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY[ + stacked_spine_module.US_LATE_PRIMARY_PUF_STAGE + ] + initial = _fill_late_contract_surface( + _stacked_gap_fixture(), + contracts=(contract,), + include_outputs=False, + ) + resources = stacked_spine_module.stacked_late_primary_resource_receipts( + pd.DataFrame({"income": [10.0]}), + primary_qrf_checkpoint_identity_sha256="a" * 64, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + seed=0, + n_estimators=100, + fit_records_enabled=True, + tail_bound_diagnostics_enabled=True, + ) + shallow = resources["tax_unit.@puf_donor_tax_units"] + shallow["binding"] = { + "resource_kind": "puf_donor_tax_units", + "schema_version": 1, + } + shallow["binding_sha256"] = stacked_spine_module._canonical_sha256( + shallow["binding"] + ) + + with pytest.raises( + ValueError, + match=( + r"(?s)primary_puf_qrf.*@effective:puf_donor.*" + r"post_clone_input_surface" + ), + ): + stacked_spine_module.run_stacked_late_producer_dag( + initial, + primary_puf_producer=lambda _frame: pytest.fail( + "shallow resource receipt reached primary callback" + ), + primary_resource_receipts=resources, + ) + + +def test_stacked_primary_qrf_refuses_stale_bound_checkpoint( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + checkpoint_identity = "a" * 64 + checkpoint_dir = tmp_path / checkpoint_identity + donor = pd.DataFrame({"fixture_donor": [1.0]}) + + def initialize(_frame: Frame, _donor: pd.DataFrame, root: Path, **_kwargs) -> None: + root.mkdir(parents=True) + (root / stacked_spine_module.PRIMARY_QRF_MANIFEST_FILENAME).write_text( + "{}", + encoding="utf-8", + ) + + monkeypatch.setattr( + stacked_spine_module, + "initialize_primary_puf_qrf_chain", + initialize, + ) + monkeypatch.setattr( + stacked_spine_module, + "primary_puf_qrf_recipient_predictor_universe_receipt", + lambda _root: {"fixture": "recipient-universe"}, + ) + monkeypatch.setattr( + stacked_spine_module, + "run_primary_puf_qrf_chain", + lambda _root, **_kwargs: None, + ) + monkeypatch.setattr( + stacked_spine_module, + "finalize_primary_puf_qrf_chain", + lambda frame, _root, **_kwargs: ( + frame, + frame.resolve_weights("tax_unit").kind, + ), + ) + + def binding(bound_donor: pd.DataFrame) -> dict[str, object]: + resources = stacked_spine_module.stacked_late_primary_resource_receipts( + bound_donor, + primary_qrf_checkpoint_identity_sha256=checkpoint_identity, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + seed=0, + n_estimators=100, + fit_records_enabled=True, + tail_bound_diagnostics_enabled=True, + ) + return stacked_spine_module.stacked_late_primary_checkpoint_input_binding( + resources + ) + + with pytest.raises( + ValueError, + match=r"checkpoint directory identity differs.*directory='b{64}'.*bound='a{64}'", + ): + stacked_spine_module._run_stacked_puf_pass_without_tail_for_test( + _late_primary_entry(_stacked_gap_fixture()), + donor, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + fit_records=[], + tail_bound_diagnostics=[], + primary_qrf_checkpoint_dir=tmp_path / ("b" * 64), + primary_qrf_input_binding=binding(donor), + ) + + stacked_spine_module._run_stacked_puf_pass_without_tail_for_test( + _late_primary_entry(_stacked_gap_fixture()), + donor, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + fit_records=[], + tail_bound_diagnostics=[], + primary_qrf_checkpoint_dir=checkpoint_dir, + primary_qrf_input_binding=binding(donor), + ) + assert (checkpoint_dir / "late-producer-input-binding.json").is_file() + + changed_donor = donor.copy() + changed_donor.iloc[0, 0] = 2.0 + with pytest.raises(ValueError, match="refusing stale predictions"): + stacked_spine_module._run_stacked_puf_pass_without_tail_for_test( + _late_primary_entry(_stacked_gap_fixture()), + changed_donor, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + fit_records=[], + tail_bound_diagnostics=[], + primary_qrf_checkpoint_dir=checkpoint_dir, + primary_qrf_input_binding=binding(changed_donor), + ) + + +def test_late_transfer_rejects_identityless_bank_before_dispatch() -> None: + group = stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS[0] + + with pytest.raises(ValueError, match="target-bank identity"): + stacked_spine_module._late_transfer_resource_receipts( + group_name=group.name, + entity=group.entity, + family=group.family, + targets=group.targets, + seed=0, + n_estimators=100, + max_targets_per_fit=( + stacked_spine_module.DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT + ), + target_bank=object(), + ) + + +def test_late_transfer_resources_bind_all_callback_controls() -> None: + group = stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS[0] + resources = stacked_spine_module._late_transfer_resource_receipts( + group_name=group.name, + entity=group.entity, + family=group.family, + targets=group.targets, + seed=0, + n_estimators=100, + max_targets_per_fit=( + stacked_spine_module.DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT + ), + target_bank=None, + ) + model = resources[f"{group.entity}.@late_transfer_model_config"]["binding"] + + assert model["donor_spine"] == stacked_spine_module.ASEC_PUF_DONOR_SPINE + assert model["donor_channel"] is None + assert model["donor_selection"] == ("all_rows_from_post_puf_asec_origin_projection") + assert model["donor_projection"] == { + "support_channel": stacked_spine_module.BASE_ASEC_SUPPORT_CHANNEL, + "support_clone_index": stacked_spine_module.PUF_TAX_DETAIL_CLONE_INDEX, + } + assert model["schema_version"] == 3 + assert model["transfer_execution_contract"] == ( + acs_transfer_module.acs_transfer_execution_contract_identity( + targets=group.targets, + derive_schedule_d=False, + ) + ) + assert model["transfer_execution_contract"]["post_transfer_structure"][ + "schedule_d_capital_gain_distributions" + ] == { + "enabled": False, + "source": "long_term_capital_gains_before_response", + "exclusive_with": "non_sch_d_capital_gains", + "output": "schedule_d_capital_gain_distributions", + "preserve_preexisting_nonnull": True, + "share_asset": None, + } + + +def test_late_transfer_refuses_a_stale_bound_execution_contract( + monkeypatch: pytest.MonkeyPatch, +) -> None: + group = next( + item + for item in stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS + if item.name == "transfer:person/adult_care" + ) + bound = acs_transfer_module.acs_transfer_execution_contract_identity( + targets=group.targets, + derive_schedule_d=False, + ) + changed_codes = dict(acs_transfer_module._TENURE_CODES) + changed_codes["OWN"] = 9.0 + monkeypatch.setattr(acs_transfer_module, "_TENURE_CODES", changed_codes) + + with pytest.raises( + ValueError, + match="runtime execution contract differs from its bound input", + ): + acs_transfer_module.transfer_acs_inputs( + _post_puf_transfer_fixture(), + _post_puf_transfer_fixture(), + target_families=group.target_families, + donor_spine=stacked_spine_module.ASEC_PUF_DONOR_SPINE, + donor_channel=None, + derive_schedule_d=False, + execution_contract=bound, + ) + + +def test_late_source_resources_bind_all_callback_controls(tmp_path: Path) -> None: + allow_existing_operators = { + "with_us_child_support_inputs", + "with_us_disability_benefits", + "with_us_workers_compensation", + "with_us_childcare_inputs", + "with_us_adult_care_inputs", + "with_us_energy_subsidy_input", + } + for operator in multispine_pool_module.POOL_POST_CLONE_SOURCE_OPERATOR_ORDER: + producer = f"source:{operator}" + resources = stacked_spine_module._late_source_resource_receipts( + producer_name=producer + ) + assert set(resources) == {"person.@post_clone_source_execution_config"} + binding = resources["person.@post_clone_source_execution_config"]["binding"] + assert binding["schema_version"] == 3 + assert binding["operator"] == operator + assert binding["phase"] == multispine_pool_module._POST_CLONE_PHASE + assert binding["operator_registry"] == list( + multispine_pool_module.POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + ) + contract = multispine_pool_module.POOL_OPERATOR_CONTRACTS[operator] + assert binding["operator_contract"] == { + "family": contract.family, + "phases": list(contract.phases), + "mechanism": contract.mechanism, + "execution_scope": contract.execution_scope, + } + assert binding["declared_output_family"] + assert binding["seed"] == multispine_pool_module.POOL_RANDOM_SEED + assert binding["time_period"] == ( + None + if operator == "impute_us_housing_assistance_to_puf_support" + else multispine_pool_module.POOL_TIME_PERIOD + ) + assert binding["force_puf_imputation"] is ( + True if operator == "with_us_retirement_distribution_inputs" else None + ) + expected_sidecars = {} + if operator == "with_us_weeks_unemployed": + expected_sidecars = {"asec_2023_source": {"mode": "not_supplied"}} + if operator == "with_us_education_inputs": + expected_sidecars = {"asec_education_source": {"mode": "not_supplied"}} + assert binding["external_sidecars"] == expected_sidecars + assert binding["allow_existing_without_source"] is ( + multispine_pool_module.POOL_SOURCE_ALLOW_EXISTING_WITHOUT_SOURCE + if operator in allow_existing_operators + else None + ) + assert binding["housing_assistance_qrf"] == ( + { + "n_estimators": multispine_pool_module.POOL_HOUSING_ASSISTANCE_N_ESTIMATORS, + "max_train_samples": ( + multispine_pool_module.POOL_HOUSING_ASSISTANCE_MAX_TRAIN_SAMPLES + ), + } + if operator == "impute_us_housing_assistance_to_puf_support" + else None + ) + source_stage = binding["source_stage_spec"] + if operator == "impute_us_housing_assistance_to_puf_support": + assert source_stage is None + else: + assert ( + source_stage["stage_name"] + == (source_stage["resolved_stage_spec"]["stage"]) + ) + assert source_stage["runtime_stage_spec_verified"] is True + assert set(source_stage["runtime_stage_spec_resolver"]) == { + "module", + "callable", + } + assert source_stage["resolved_stage_spec_sha256"] == ( + stacked_spine_module._canonical_sha256( + source_stage["resolved_stage_spec"] + ) + ) + + source_asset = stacked_spine_module.files("microcosm.build.us").joinpath( + "source_stages.json" + ) + changed_payload = json.loads(source_asset.read_text(encoding="utf-8")) + stage = next( + item + for item in changed_payload["stages"] + if item["stage"] == "adult_care_inputs" + ) + stage["notes"] = f"{stage.get('notes', '')} identity-regression" + changed_asset = tmp_path / "source_stages.json" + changed_asset.write_text(json.dumps(changed_payload), encoding="utf-8") + baseline = stacked_spine_module._late_source_stage_spec_binding( + "with_us_adult_care_inputs" + ) + changed = stacked_spine_module._late_source_stage_spec_binding( + "with_us_adult_care_inputs", + resource=changed_asset, + ) + assert baseline["asset_sha256"] != changed["asset_sha256"] + assert ( + baseline["resolved_stage_spec_sha256"] + != (changed["resolved_stage_spec_sha256"]) + ) + + +def test_source_resource_refuses_live_callback_control_drift( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(multispine_pool_module, "POOL_RANDOM_SEED", 913) + + with pytest.raises(ValueError, match="late source execution config changed"): + stacked_spine_module._late_source_resource_receipts( + producer_name="source:with_us_adult_care_inputs" + ) + + +def test_source_resource_binding_does_not_introspect_injected_runner( + monkeypatch: pytest.MonkeyPatch, +) -> None: + def injected_runner(*_args: object, **_kwargs: object) -> PoolStageOutput: + raise AssertionError("resource construction must not execute the runner") + + monkeypatch.setattr( + multispine_pool_module, + "_run_source_operator_chain", + injected_runner, + ) + resources = stacked_spine_module._late_source_resource_receipts( + producer_name="source:with_us_adult_care_inputs" + ) + binding = resources["person.@post_clone_source_execution_config"]["binding"] + family = multispine_pool_module.POOL_OPERATOR_CONTRACTS[ + "with_us_adult_care_inputs" + ].family + assert binding["declared_output_family"] == { + entity: sorted(columns) + for entity, columns in sorted( + multispine_pool_module.PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES[family].items() + ) + } + + +def test_source_resource_refuses_runtime_stage_helper_drift( + monkeypatch: pytest.MonkeyPatch, +) -> None: + runtime = stacked_spine_module.importlib.import_module( + "microcosm.build.us_runtime.adult_care" + ) + stage_spec = runtime.us_adult_care_stage_spec() + monkeypatch.setattr( + runtime, + "us_adult_care_stage_spec", + lambda: replace(stage_spec, notes=f"{stage_spec.notes} drift"), + ) + + with pytest.raises(ValueError, match="runtime SourceStageSpec differs"): + stacked_spine_module._late_source_resource_receipts( + producer_name="source:with_us_adult_care_inputs" + ) + + +def test_late_source_finalizer_resources_bind_all_callback_controls() -> None: + resources = stacked_spine_module._late_source_finalizer_resource_receipts() + assert set(resources) == {"person.@source_finalizer_execution_config"} + binding = resources["person.@source_finalizer_execution_config"]["binding"] + assert binding["schema_version"] == 2 + assert binding == stacked_spine_module._late_source_finalizer_execution_binding() + assert binding["source_operator_registry"] == list( + multispine_pool_module.POOL_POST_CLONE_SOURCE_OPERATOR_ORDER + ) + assert binding["deferred_transfer_inputs"] == ( + multispine_pool_module.POOL_DEFERRED_TRANSFER_INPUTS + ) + + +def test_source_finalizer_resource_refuses_live_doctrine_drift( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr( + multispine_pool_module, + "POOL_DEFERRED_TRANSFER_STATUS", + "unreceipted_runtime_drift", + ) + + with pytest.raises(ValueError, match="late source-finalizer config changed"): + stacked_spine_module._late_source_finalizer_resource_receipts() + + +def test_primary_refuses_missing_universe_receipt_before_callback() -> None: + contract = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY[ + stacked_spine_module.US_LATE_PRIMARY_PUF_STAGE + ] + initial = _fill_late_contract_surface( + _stacked_gap_fixture(), + contracts=(contract,), + include_outputs=False, + ) + resources = stacked_spine_module.stacked_late_primary_resource_receipts( + pd.DataFrame({"fixture_donor": [1.0]}), + primary_qrf_checkpoint_identity_sha256="a" * 64, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + seed=0, + n_estimators=100, + fit_records_enabled=True, + tail_bound_diagnostics_enabled=True, + ) + unfilled, invalid = stacked_spine_module._late_input_readiness_rows( + initial, + contract, + available_input_receipts=resources, + ) + + with pytest.raises( + ValueError, + match=( + r"(?s)primary_puf_qrf.*" + r"frame\.@acs_pums_earnings_universe_application.*1 unfilled.*" + r"acs_pums_earnings_universe" + ), + ): + stacked_spine_module.run_producer_when_ready( + contract, + lambda: pytest.fail("universe-less frame reached primary callback"), + unfilled_rows=unfilled, + invalid_rows=invalid, + absence_receipts={}, + ) + + +def test_primary_tax_unit_passthrough_requires_finite_or_declared_absence() -> None: + contract = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY[ + stacked_spine_module.US_LATE_PRIMARY_PUF_STAGE + ] + initial = _fill_late_contract_surface( + _late_primary_entry(_stacked_gap_fixture()), + contracts=(contract,), + include_outputs=False, + ) + column = stacked_spine_module.PUF_TAX_DETAIL_DEFAULT_TAX_UNIT_OUTPUTS[0] + requirement = next( + item + for item in contract.inputs + if item.column == f"@effective:tax_unit_output_passthrough:{column}" + ) + resources = stacked_spine_module.stacked_late_primary_resource_receipts( + pd.DataFrame({"fixture_donor": [1.0]}), + primary_qrf_checkpoint_identity_sha256="a" * 64, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + seed=0, + n_estimators=100, + fit_records_enabled=True, + tail_bound_diagnostics_enabled=True, + ) + + absent_tables = { + entity: initial.table(entity).copy() for entity in initial.entities + } + absent_tables["tax_unit"].drop(columns=column, inplace=True) + absent = Frame( + absent_tables, + initial.schema, + {entity: initial.weights_for(entity) for entity in initial.weighted_entities}, + initial.strata, + mass_log=initial.mass_log, + metadata=initial.metadata, + ) + unfilled, invalid = stacked_spine_module._late_input_readiness_rows( + absent, + contract, + available_input_receipts=resources, + ) + absence_receipts = stacked_spine_module._late_declared_absence_receipts( + contract, + unfilled, + invalid_rows=invalid, + ) + receipt_id = requirement.tolerated_absence_receipts[0] + assert unfilled[requirement] == len(absent.table("tax_unit")) + assert invalid[requirement] == 0 + assert absence_receipts[receipt_id]["status"] == "declared_absence" + assert ( + stacked_spine_module.run_producer_when_ready( + contract, + lambda: "ran", + unfilled_rows=unfilled, + invalid_rows=invalid, + absence_receipts=absence_receipts, + ) + == "ran" + ) + + invalid_tables = { + entity: initial.table(entity).copy() for entity in initial.entities + } + invalid_tables["tax_unit"][column] = invalid_tables["tax_unit"][column].astype( + object + ) + invalid_tables["tax_unit"].loc[invalid_tables["tax_unit"].index[0], column] = ( + "not-numeric" + ) + invalid_frame = Frame( + invalid_tables, + initial.schema, + {entity: initial.weights_for(entity) for entity in initial.weighted_entities}, + initial.strata, + mass_log=initial.mass_log, + metadata=initial.metadata, + ) + unfilled, invalid = stacked_spine_module._late_input_readiness_rows( + invalid_frame, + contract, + available_input_receipts=resources, + ) + absence_receipts = stacked_spine_module._late_declared_absence_receipts( + contract, + unfilled, + invalid_rows=invalid, + ) + assert unfilled[requirement] == 0 + assert invalid[requirement] == 1 + assert receipt_id not in absence_receipts + with pytest.raises( + ValueError, + match=( + rf"(?s)primary_puf_qrf.*tax_unit_output_passthrough:{column}.*" + r"1 invalid.*post_clone_input_surface" + ), + ): + stacked_spine_module.run_producer_when_ready( + contract, + lambda: pytest.fail("invalid tax-unit passthrough reached callback"), + unfilled_rows=unfilled, + invalid_rows=invalid, + absence_receipts=absence_receipts, + ) + + +def test_universe_resource_binds_exact_contract_and_scope() -> None: + resources = stacked_spine_module._late_acs_earnings_universe_resource_receipts() + receipt = resources["person.@acs_pums_earnings_universe_execution_config"] + binding = receipt["binding"] + assert binding["schema_version"] == 2 + assert binding["runtime_identity_owner"] == ( + "microcosm.build.us_runtime.acs_income_universe" + ) + assert binding["ordered_mapped_columns"] == [ + "employment_income_before_lsr", + "self_employment_income_before_lsr", + ] + assert binding["person_scope_mode"] == "whole_frame_acs_channel" + assert binding["contract_identity"] == ( + stacked_spine_module.acs_pums_earnings_universe_contract_identity() + ) + + +def test_universe_resource_refuses_live_contract_drift( + monkeypatch: pytest.MonkeyPatch, +) -> None: + changed = deepcopy(universe_module.acs_pums_earnings_universe_contract_identity()) + changed["minimum_age"] = 14 + monkeypatch.setattr( + universe_module, + "acs_pums_earnings_universe_contract_identity", + lambda: changed, + ) + + with pytest.raises(ValueError, match="late ACS earnings-universe contract changed"): + stacked_spine_module._late_acs_earnings_universe_resource_receipts() + + +def _late_universe_entry_fixture() -> Frame: + registry = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY + primary_contract = registry[stacked_spine_module.US_LATE_PRIMARY_PUF_STAGE] + initial = _fill_late_contract_surface( + _stacked_gap_fixture(), + contracts=(primary_contract,), + include_outputs=False, + ) + initial_person = initial.table("person") + structural_row = initial_person.index[ + initial_person[support_channel_column("person")].eq("acs") + ][0] + initial_person.loc[structural_row, "age"] = 12.0 + initial_person.loc[ + structural_row, + [ + "WAGP", + "SEMP", + "employment_income_before_lsr", + "self_employment_income_before_lsr", + ], + ] = np.nan + return initial + + +def test_universe_raw_authority_binds_present_column_with_structural_nulls() -> None: + contract = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY[ + stacked_spine_module.US_LATE_ACS_EARNINGS_UNIVERSE_STAGE + ] + resources = stacked_spine_module._late_acs_earnings_universe_resource_receipts() + initial = _late_universe_entry_fixture() + unfilled, invalid = stacked_spine_module._late_input_readiness_rows( + initial, + contract, + available_input_receipts=resources, + ) + wagp = next( + requirement + for requirement in contract.inputs + if requirement.column == "@effective:raw_source:WAGP" + ) + assert unfilled[wagp] == 0 + assert invalid[wagp] == 0 + + evidence = stacked_spine_module._late_declared_input_evidence( + initial, + contract, + available_input_receipts=resources, + unfilled_rows=unfilled, + invalid_rows=invalid, + ) + baseline_sha256 = stacked_spine_module._canonical_sha256(evidence) + + changed_person = initial.table("person").copy() + eligible = changed_person[support_channel_column("person")].eq( + "acs" + ) & pd.to_numeric(changed_person["age"], errors="raise").ge(15) + changed_person.loc[changed_person.index[eligible][0], "WAGP"] = 2.0 + changed = Frame( + { + entity: changed_person if entity == "person" else initial.table(entity) + for entity in initial.entities + }, + initial.schema, + {entity: initial.weights_for(entity) for entity in initial.weighted_entities}, + initial.strata, + mass_log=initial.mass_log, + metadata=initial.metadata, + ) + changed_unfilled, changed_invalid = stacked_spine_module._late_input_readiness_rows( + changed, + contract, + available_input_receipts=resources, + ) + changed_evidence = stacked_spine_module._late_declared_input_evidence( + changed, + contract, + available_input_receipts=resources, + unfilled_rows=changed_unfilled, + invalid_rows=changed_invalid, + ) + assert stacked_spine_module._canonical_sha256(changed_evidence) != baseline_sha256 + + absent_person = initial.table("person").drop(columns=["WAGP"]) + absent = Frame( + { + entity: absent_person if entity == "person" else initial.table(entity) + for entity in initial.entities + }, + initial.schema, + {entity: initial.weights_for(entity) for entity in initial.weighted_entities}, + initial.strata, + mass_log=initial.mass_log, + metadata=initial.metadata, + ) + absent_unfilled, absent_invalid = stacked_spine_module._late_input_readiness_rows( + absent, + contract, + available_input_receipts=resources, + ) + assert absent_unfilled[wagp] > 0 + with pytest.raises( + ValueError, + match=r"(?s)acs_pums_earnings_universe.*raw_source:WAGP.*post_clone_input_surface", + ): + stacked_spine_module.run_producer_when_ready( + contract, + lambda: pytest.fail("missing WAGP reached universe callback"), + unfilled_rows=absent_unfilled, + invalid_rows=absent_invalid, + absence_receipts={}, + ) + + +def _run_real_late_executor_fixture( + monkeypatch: pytest.MonkeyPatch, + *, + bank_identity_sha256: str | None = None, + bound_clone_attachment_seed: int = 578, + asec_earnings_delta: float = 0.0, +) -> tuple[stacked_spine_module.StackedLateProducerResult, tuple[str, ...], int]: + registry = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY + initial = _late_universe_entry_fixture() + initial_person = initial.table("person") + if asec_earnings_delta: + asec_row = initial_person.index[ + initial_person[support_channel_column("person")].eq("asec") + ][0] + initial_person.loc[asec_row, "employment_income_before_lsr"] += ( + asec_earnings_delta + ) + events: list[str] = [] + finalizer_calls = 0 + + materialize_universe = ( + stacked_spine_module._materialize_stacked_acs_earnings_universe + ) + + def universe( + frame: Frame, + *, + execution_config: Mapping[str, object] | None = None, + ): + events.append(stacked_spine_module.US_LATE_ACS_EARNINGS_UNIVERSE_STAGE) + assert execution_config is not None + return materialize_universe(frame, execution_config=execution_config) + + donor = pd.DataFrame({"fixture_donor": [1.0]}) + actual_primary_resources = ( + stacked_spine_module.stacked_late_primary_resource_receipts( + donor, + primary_qrf_checkpoint_identity_sha256="a" * 64, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + seed=0, + n_estimators=100, + fit_records_enabled=True, + tail_bound_diagnostics_enabled=True, + ) + ) + + def primary(frame: Frame): + events.append(stacked_spine_module.US_LATE_PRIMARY_PUF_STAGE) + attached = clone_us_frame_for_puf_support( + frame, + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + ) + attached = Frame( + {entity: attached.table(entity) for entity in attached.entities}, + attached.schema, + { + entity: attached.weights_for(entity) + for entity in attached.weighted_entities + }, + attached.strata, + mass_log=attached.mass_log, + metadata={ + **attached.metadata, + "us_puf_clone_attachment_manifest": {"fixture": True}, + }, + ) + completed = _fill_late_contract_surface( + attached, + contracts=tuple(registry.values()), + include_outputs=True, + ) + return stacked_spine_module.StackedPufPassResult( + completed, + { + "primary_resource_receipts_sha256": ( + stacked_spine_module._canonical_sha256(actual_primary_resources) + ) + }, + ) + + def source(frame: Frame, operator: str) -> PoolStageOutput: + events.append(f"source:{operator}") + return PoolStageOutput( + frame, + { + "phase": "post_clone", + "operator_order": [operator], + "suboperators": [{"operator": operator}], + }, + ) + + def finalize( + frame: Frame, + *, + operator_receipts: dict[str, object], + ) -> PoolStageOutput: + nonlocal finalizer_calls + finalizer_calls += 1 + events.append(stacked_spine_module.US_LATE_SOURCE_FINALIZER_STAGE) + source_order = list(operator_receipts) + return PoolStageOutput( + frame, + { + "phase": "post_clone", + "operator_order": source_order, + "suboperators": [ + {"operator": operator, "order_index": index} + for index, operator in enumerate(source_order) + ], + "deferred_transfer_inputs": { + "inputs": { + column: {} + for column in ( + "bank_account_assets", + "bond_assets", + "stock_assets", + ) + } + }, + }, + ) + + def transfer( + frame: Frame, + *, + group_name: str, + execution_contract: Mapping[str, object] | None = None, + **_kwargs: object, + ) -> stacked_spine_module.StackedPostPufTransferResult: + events.append(group_name) + group = next( + item + for item in stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS + if item.name == group_name + ) + assert execution_contract == ( + acs_transfer_module.acs_transfer_execution_contract_identity( + targets=group.targets, + derive_schedule_d=False, + ) + ) + transfer_result = AcsTransferResult( + frame=frame, + imputed_inputs=(), + fit_records=(), + deferred_inputs=(), + resolved_donor_channel="asec", + ) + return stacked_spine_module.StackedPostPufTransferResult( + frame, + { + "producer": group.name, + "ordered_targets": list(group.targets), + "targets": { + f"{group.entity}/{group.family}/{target}": { + "residual_null_rows": 0, + } + for target in group.targets + }, + }, + transfer_result, + ) + + monkeypatch.setattr( + stacked_spine_module, + "_materialize_stacked_acs_earnings_universe", + universe, + ) + monkeypatch.setattr( + multispine_pool_module, + "run_multispine_post_clone_source_operator", + source, + ) + monkeypatch.setattr( + multispine_pool_module, + "finalize_multispine_source_inputs", + finalize, + ) + monkeypatch.setattr( + stacked_spine_module, + "transfer_stacked_post_puf_group", + transfer, + ) + resources = stacked_spine_module.stacked_late_primary_resource_receipts( + donor, + primary_qrf_checkpoint_identity_sha256="a" * 64, + clone_attachment_fraction=1.0, + clone_attachment_seed=bound_clone_attachment_seed, + seed=0, + n_estimators=100, + fit_records_enabled=True, + tail_bound_diagnostics_enabled=True, + ) + target_banks = None + if bank_identity_sha256 is not None: + + class IdentityBank: + identity_sha256 = bank_identity_sha256 + + target_banks = { + group.name: IdentityBank() + for group in stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS + } + + result = stacked_spine_module.run_stacked_late_producer_dag( + initial, + primary_puf_producer=primary, + primary_resource_receipts=resources, + target_banks=target_banks, + ) + + return result, tuple(events), finalizer_calls + + +def test_real_late_executor_follows_canonical_order_and_finalizes_sources_once( + monkeypatch: pytest.MonkeyPatch, +) -> None: + result, events, finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + schedule = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_SCHEDULE + + assert events == schedule.order + assert finalizer_calls == 1 + universe_row = result.receipt["execution"][0] + assert universe_row["producer"] == ( + stacked_spine_module.US_LATE_ACS_EARNINGS_UNIVERSE_STAGE + ) + assert universe_row["declared_absence_receipts"] + produced_person = result.primary_puf_result.frame.table("person") + produced_child = produced_person[ + produced_person[support_channel_column("person")].eq("acs") + & produced_person["age"].lt(15) + ] + assert ( + produced_child[ + [ + "employment_income_before_lsr", + "self_employment_income_before_lsr", + ] + ] + .eq(0.0) + .all() + .all() + ) + assert events.index("transfer:person/puf_tax_itemization__batch_5") < events.index( + "source:with_us_adult_care_inputs" + ) + assert ( + result.transition_authority_sha256 + == result.frame.metadata[ + stacked_spine_module.US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY + ]["sha256"] + ) + stacked_spine_module.validate_stacked_late_producer_receipt( + result.receipt, + boundary="executor regression", + frame=result.frame, + expected_transition_authority_sha256=result.transition_authority_sha256, + ) + + +def test_late_executor_authority_binds_every_transfer_bank_identity( + monkeypatch: pytest.MonkeyPatch, +) -> None: + first, _events, _finalizer_calls = _run_real_late_executor_fixture( + monkeypatch, + bank_identity_sha256="a" * 64, + ) + second, _events, _finalizer_calls = _run_real_late_executor_fixture( + monkeypatch, + bank_identity_sha256="b" * 64, + ) + assert first.transition_authority_sha256 != second.transition_authority_sha256 + transfer_rows = [ + row for row in first.receipt["execution"] if row["kind"] == "late_transfer" + ] + assert len(transfer_rows) == 19 + for row in transfer_rows: + available = row["available_input_receipts"] + assert len(available) == 2 + bank = next( + receipt + for key, receipt in available.items() + if key.endswith(".@late_transfer_target_bank") + ) + assert bank["binding"] == { + "resource_kind": "late_transfer_target_bank", + "schema_version": 1, + "mode": "identity_bound_checkpoint", + "identity_sha256": "a" * 64, + } + + +def test_late_executor_rejects_primary_callback_resource_mismatch( + monkeypatch: pytest.MonkeyPatch, +) -> None: + with pytest.raises( + ValueError, + match="primary callback receipt disagrees with its declared resources", + ): + _run_real_late_executor_fixture( + monkeypatch, + bound_clone_attachment_seed=579, + ) + + +def test_universe_receipt_excludes_out_of_scope_asec_earnings_values( + monkeypatch: pytest.MonkeyPatch, +) -> None: + baseline, _events, _finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + changed, _events, _finalizer_calls = _run_real_late_executor_fixture( + monkeypatch, + asec_earnings_delta=1.0, ) - with pytest.raises(TypeError, match="semantic strings cannot mix"): - stacked_spine_module._canonical_donor_series_payload( - mixed, - boundary="mixed-object donor identity", - ) + baseline_row = baseline.receipt["execution"][0] + changed_row = changed.receipt["execution"][0] + assert baseline_row["input_surface_sha256"] == changed_row["input_surface_sha256"] + assert ( + baseline_row["producer_receipt_sha256"] + == changed_row["producer_receipt_sha256"] + ) -def test_donor_byte_identity_ignores_string_object_alias_topology() -> None: - shared = "".join(["dynamically", "-", "allocated", "-", "donor"]) - separate = [ - "".join(["dynamically", "-", "allocated", "-", "donor"]) for _ in range(2) + +@pytest.mark.parametrize( + "column", + ( + "person_tax_unit_id", + "person_support_clone_index", + "person_source_id", + ), +) +def test_universe_receipt_affecting_acs_identity_changes_input_surface( + column: str, +) -> None: + contract = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY[ + stacked_spine_module.US_LATE_ACS_EARNINGS_UNIVERSE_STAGE ] - before = pd.Series([shared, shared], name="native_label", dtype=object) - after = pd.Series(separate, name="native_label", dtype=object) - assert before.iloc[0] is before.iloc[1] - assert after.iloc[0] is not after.iloc[1] + resources = stacked_spine_module._late_acs_earnings_universe_resource_receipts() - assert stacked_spine_module._canonical_donor_series_payload( - before, - boundary="aliased-string donor identity before", - ) == stacked_spine_module._canonical_donor_series_payload( - after, - boundary="aliased-string donor identity after", - ) + def identities(frame: Frame) -> tuple[str, str]: + unfilled, invalid = stacked_spine_module._late_input_readiness_rows( + frame, + contract, + available_input_receipts=resources, + ) + evidence = stacked_spine_module._late_declared_input_evidence( + frame, + contract, + available_input_receipts=resources, + unfilled_rows=unfilled, + invalid_rows=invalid, + ) + application = stacked_spine_module._materialize_stacked_acs_earnings_universe( + frame + ) + return ( + stacked_spine_module._canonical_sha256(evidence), + stacked_spine_module._canonical_sha256(application.receipt), + ) + baseline = _late_universe_entry_fixture() + changed = _late_universe_entry_fixture() + person = changed.table("person") + structural_row = person.index[ + person[support_channel_column("person")].eq("acs") & person["age"].lt(15) + ][0] + if column == "person_tax_unit_id": + previous_id = int(person.loc[structural_row, column]) + replacement_id = previous_id - 1 + person.loc[person[column].eq(previous_id), column] = replacement_id + tax_unit = changed.table("tax_unit") + tax_unit.loc[tax_unit["tax_unit_id"].eq(previous_id), "tax_unit_id"] = ( + replacement_id + ) + else: + person.loc[structural_row, column] = ( + int(person.loc[structural_row, column]) + 10_000 + ) -def test_donor_byte_identity_accepts_semantic_boolean_object_scalars() -> None: - semantic_booleans = pd.Series( - [True, np.bool_(False), None], - name="is_disabled", - dtype=object, - ) + baseline_input, baseline_receipt = identities(baseline) + changed_input, changed_receipt = identities(changed) + assert changed_input != baseline_input + assert changed_receipt != baseline_receipt - assert stacked_spine_module._canonical_donor_series_payload( - semantic_booleans, - boundary="semantic-boolean donor identity before", - ) == stacked_spine_module._canonical_donor_series_payload( - semantic_booleans.copy(deep=True), - boundary="semantic-boolean donor identity after", + +def test_late_receipt_rejects_internally_consistent_forgery_against_authority( + monkeypatch: pytest.MonkeyPatch, +) -> None: + result, _events, _finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + forged = deepcopy(dict(result.receipt)) + forged["input_frame_sha256"] = "0" * 64 + previous = stacked_spine_module._late_execution_genesis_sha256( + producer_schedule_sha256=forged["producer_schedule"]["payload_sha256"], + input_frame_sha256=forged["input_frame_sha256"], + ) + for row in forged["execution"]: + row["previous_execution_sha256"] = previous + row.pop("sha256") + row["sha256"] = stacked_spine_module._canonical_sha256(row) + previous = row["sha256"] + forged["execution_chain_sha256"] = previous + forged.pop("sha256") + forged["sha256"] = stacked_spine_module._canonical_sha256(forged) + forged_authority = stacked_spine_module._late_producer_transition_authority_receipt( + forged + ) + forged_frame = Frame( + {entity: result.frame.table(entity) for entity in result.frame.entities}, + result.frame.schema, + { + entity: result.frame.weights_for(entity) + for entity in result.frame.weighted_entities + }, + result.frame.strata, + mass_log=result.frame.mass_log, + metadata={ + **result.frame.metadata, + stacked_spine_module.US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY: ( + forged_authority + ), + }, ) + with pytest.raises( + ValueError, + match="independently carried late-producer transition authority", + ): + stacked_spine_module.validate_stacked_late_producer_receipt( + forged, + boundary="forged executor regression", + frame=forged_frame, + expected_transition_authority_sha256=(result.transition_authority_sha256), + ) -def test_gap_fill_activation_authority_fails_closed_on_donor_nulls() -> None: - stacked = _stacked_gap_fixture() - person = stacked.table("person").copy() - channel = person[support_channel_column("person")] - poke = person.index[channel.eq("asec")][1] - person.loc[poke, "unemployment_compensation"] = np.nan - tables = {entity: stacked.table(entity) for entity in stacked.entities} + +def test_late_receipt_rejects_live_output_content_drift( + monkeypatch: pytest.MonkeyPatch, +) -> None: + result, _events, _finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + tables = {entity: result.frame.table(entity) for entity in result.frame.entities} + person = tables["person"].copy() + target = "sstb_self_employment_income_before_lsr" + person.loc[person.index[0], target] = float(person.loc[person.index[0], target]) + 1 tables["person"] = person - poked = Frame( + drifted = Frame( tables, - stacked.schema, - {entity: stacked.weights_for(entity) for entity in stacked.weighted_entities}, - stacked.strata, - mass_log=stacked.mass_log, - metadata=stacked.metadata, + result.frame.schema, + { + entity: result.frame.weights_for(entity) + for entity in result.frame.weighted_entities + }, + result.frame.strata, + mass_log=result.frame.mass_log, + metadata=result.frame.metadata, ) - with pytest.raises(ValueError, match="donors must observe"): - _gap_fill_with_test_authority(poked, plan=_GAP_FILL_TEST_PLAN, seed=578) + with pytest.raises(ValueError, match="output digest does not match the live frame"): + stacked_spine_module.validate_stacked_late_producer_receipt( + result.receipt, + boundary="drifted executor regression", + frame=drifted, + expected_transition_authority_sha256=(result.transition_authority_sha256), + ) -@pytest.mark.parametrize( - ("recipient_channel", "donor_channel", "role", "missing_channel"), - ( - ("acx_typo", "asec", "recipient", "acx_typo"), - ("acs", "asec_typo", "donor", "asec_typo"), - ), -) -def test_gap_fill_activation_authority_rejects_nonlive_declared_channel( - recipient_channel: str, - donor_channel: str, - role: str, - missing_channel: str, +def _rehash_late_receipt_after_fixture_mutation( + receipt: dict[str, object], ) -> None: - plan = ( - GapFillDirection( - name="nonlive_channel", - recipient_channel=recipient_channel, - donor_channel=donor_channel, - target_families={"person": {"complete": ("age",)}}, - ), + previous = stacked_spine_module._late_execution_genesis_sha256( + producer_schedule_sha256=receipt["producer_schedule"]["payload_sha256"], + input_frame_sha256=receipt["input_frame_sha256"], ) + for row in receipt["execution"]: + row["input_surface_sha256"] = stacked_spine_module._canonical_sha256( + row["declared_inputs"] + ) + row["previous_execution_sha256"] = previous + row.pop("sha256", None) + row["sha256"] = stacked_spine_module._canonical_sha256(row) + previous = row["sha256"] + receipt["execution_chain_sha256"] = previous + receipt.pop("sha256", None) + receipt["sha256"] = stacked_spine_module._canonical_sha256(receipt) + + +def test_late_receipt_rejects_forged_absent_required_virtual_input( + monkeypatch: pytest.MonkeyPatch, +) -> None: + result, _events, _finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + forged = deepcopy(dict(result.receipt)) + primary = next( + row + for row in forged["execution"] + if row["producer"] == stacked_spine_module.US_LATE_PRIMARY_PUF_STAGE + ) + config_key = "tax_unit.@primary_puf_execution_config" + config_input = next( + item + for item in primary["declared_inputs"] + if item["column"] == "@effective:primary_puf_execution_config" + ) + config_evidence = config_input["evidence"] + config_column = config_evidence["alternatives"][0][0] + config_column["status"] = "absent" + config_column["content_sha256"] = stacked_spine_module._canonical_sha256( + {"absent": True} + ) + config_evidence["sha256"] = stacked_spine_module._canonical_sha256( + {"alternatives": config_evidence["alternatives"]} + ) + del primary["available_input_receipts"][config_key] + _rehash_late_receipt_after_fixture_mutation(forged) with pytest.raises( ValueError, - match=f"declared {role} channel {missing_channel!r} has no live rows", + match="inconsistent counts or content identity", ): - _gap_fill_with_test_authority(_stacked_gap_fixture(), plan=plan, seed=578) + stacked_spine_module.validate_stacked_late_producer_receipt( + forged, + boundary="forged required virtual input", + ) -def test_gap_fill_fails_closed_on_missing_target_column() -> None: - stacked = _stacked_gap_fixture() - plan = ( - GapFillDirection( - name="asec_survey_to_acs", - recipient_channel="acs", - donor_channel="asec", - target_families={ - "person": {"model_required_numeric": ("veterans_benefits",)} - }, - ), - ) - with pytest.raises(ValueError, match="veterans_benefits.*absent"): - _gap_fill_with_test_authority(stacked, plan=plan, seed=578) +def test_late_receipt_rejects_virtual_evidence_receipt_digest_disagreement( + monkeypatch: pytest.MonkeyPatch, +) -> None: + result, _events, _finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + forged = deepcopy(dict(result.receipt)) + primary = next( + row + for row in forged["execution"] + if row["producer"] == stacked_spine_module.US_LATE_PRIMARY_PUF_STAGE + ) + donor_input = next( + item + for item in primary["declared_inputs"] + if item["column"] == "@effective:puf_donor" + ) + donor_evidence = donor_input["evidence"] + donor_evidence["alternatives"][0][0]["content_sha256"] = "0" * 64 + donor_evidence["sha256"] = stacked_spine_module._canonical_sha256( + {"alternatives": donor_evidence["alternatives"]} + ) + _rehash_late_receipt_after_fixture_mutation(forged) + + with pytest.raises(ValueError, match="disagrees with its declared content"): + stacked_spine_module.validate_stacked_late_producer_receipt( + forged, + boundary="forged virtual input digest", + ) -def test_gap_fill_rejects_cloned_frames() -> None: - cloned = clone_us_frame_for_puf_support(_stacked_gap_fixture()) - with pytest.raises(ValueError, match="before clone operators"): - _gap_fill_with_test_authority(cloned, plan=_GAP_FILL_TEST_PLAN, seed=578) +def test_late_receipt_recomputes_readiness_from_physical_evidence( + monkeypatch: pytest.MonkeyPatch, +) -> None: + result, _events, _finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + forged = deepcopy(dict(result.receipt)) + primary = next( + row + for row in forged["execution"] + if row["producer"] == stacked_spine_module.US_LATE_PRIMARY_PUF_STAGE + ) + filing_status = next( + item + for item in primary["declared_inputs"] + if item["column"] == "@effective:filing_status" + ) + physical = filing_status["evidence"]["alternatives"][0][0] + physical["missing_rows"] = 1 + filing_status["evidence"]["sha256"] = stacked_spine_module._canonical_sha256( + {"alternatives": filing_status["evidence"]["alternatives"]} + ) + _rehash_late_receipt_after_fixture_mutation(forged) + + with pytest.raises(ValueError, match="readiness counts disagree"): + stacked_spine_module.validate_stacked_late_producer_receipt( + forged, + boundary="forged readiness counts", + ) -def _post_puf_transfer_fixture() -> Frame: - attached = clone_us_frame_for_puf_support( - _stacked_gap_fixture(), - clone_attachment_fraction=0.75, - clone_attachment_seed=578, - ) - person = attached.table("person").copy() - source_producer_rows = ( - person[support_channel_column("person")].astype(str).eq("asec") - ) - person["is_pregnant"] = pd.Series( - pd.NA, - index=person.index, - dtype="boolean", - ) - person.loc[source_producer_rows, "is_pregnant"] = np.resize( - np.asarray([True, False]), - int(source_producer_rows.sum()), - ) - tables = {entity: attached.table(entity) for entity in attached.entities} - tables["person"] = person - return Frame( - tables, - attached.schema, - {entity: attached.weights_for(entity) for entity in attached.weighted_entities}, - attached.strata, - mass_log=attached.mass_log, - metadata=attached.metadata, - ) +def test_late_receipt_rejects_completed_absent_output( + monkeypatch: pytest.MonkeyPatch, +) -> None: + result, _events, _finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + forged = deepcopy(dict(result.receipt)) + universe = forged["execution"][0] + universe["output_surface"][0]["status"] = "absent" + universe["output_surface_sha256"] = stacked_spine_module._canonical_sha256( + universe["output_surface"] + ) + _rehash_late_receipt_after_fixture_mutation(forged) + + with pytest.raises(ValueError, match="completed with absent output"): + stacked_spine_module.validate_stacked_late_producer_receipt( + forged, + boundary="forged absent output", + ) + + +def test_late_receipt_rejects_rehashed_output_scope_cardinality_forgery( + monkeypatch: pytest.MonkeyPatch, +) -> None: + result, _events, _finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + forged = deepcopy(dict(result.receipt)) + universe = forged["execution"][0] + output = next(item for item in universe["output_surface"] if "scope_rows" in item) + output["scope_rows"] = 0 + universe["output_surface_sha256"] = stacked_spine_module._canonical_sha256( + universe["output_surface"] + ) + _rehash_late_receipt_after_fixture_mutation(forged) + + with pytest.raises(ValueError, match="scope cardinality"): + stacked_spine_module.validate_stacked_late_producer_receipt( + forged, + boundary="forged output scope rows", + ) + + +def test_late_receipt_rejects_rehashed_duplicate_physical_input_forgery( + monkeypatch: pytest.MonkeyPatch, +) -> None: + result, _events, _finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + forged = deepcopy(dict(result.receipt)) + primary = next( + row + for row in forged["execution"] + if row["producer"] == stacked_spine_module.US_LATE_PRIMARY_PUF_STAGE + ) + hits = [ + (declared_input, column) + for declared_input in primary["declared_inputs"] + for alternative in declared_input["evidence"]["alternatives"] + for column in alternative + if column["entity"] == "person" and column["column"] == "person_id" + ] + assert len(hits) > 1 + declared_input, column = hits[1] + column["content_sha256"] = "0" * 64 + declared_input["evidence"]["sha256"] = stacked_spine_module._canonical_sha256( + {"alternatives": declared_input["evidence"]["alternatives"]} + ) + _rehash_late_receipt_after_fixture_mutation(forged) + + with pytest.raises(ValueError, match="inconsistent physical input evidence"): + stacked_spine_module.validate_stacked_late_producer_receipt( + forged, + boundary="forged duplicate input evidence", + ) + + +def test_late_receipt_rejects_detached_source_receipt_output_digest( + monkeypatch: pytest.MonkeyPatch, +) -> None: + result, _events, _finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + forged = deepcopy(dict(result.receipt)) + source = next( + row for row in forged["execution"] if row["kind"] == "post_clone_source" + ) + output = next( + item + for item in source["output_surface"] + if item["column"].startswith("@source_receipt:") + ) + output["content_sha256"] = "0" * 64 + source["output_surface_sha256"] = stacked_spine_module._canonical_sha256( + source["output_surface"] + ) + _rehash_late_receipt_after_fixture_mutation(forged) + + with pytest.raises(ValueError, match="source-receipt output digest"): + stacked_spine_module.validate_stacked_late_producer_receipt( + forged, + boundary="forged detached source receipt", + ) def test_post_puf_transfer_preserves_complete_asec_source_producers() -> None: @@ -2985,6 +5859,7 @@ def test_run_stacked_puf_pass_imputes_only_the_attached_arm() -> None: seed=578, n_estimators=10, ).frame + gap_filled = _late_primary_entry(gap_filled) donor = pd.DataFrame( { "employment_income": [45_000.0, 8_000.0, 70_000.0, 22_000.0], @@ -3063,6 +5938,7 @@ def test_run_stacked_puf_pass_receipts_raw_child_universe_application() -> None: } ) + gap_filled = _late_primary_entry(gap_filled) result = stacked_spine_module._run_stacked_puf_pass_without_tail_for_test( gap_filled, donor, @@ -3110,6 +5986,7 @@ def test_run_stacked_puf_pass_fraction_one_receipts_out_of_frame_identity() -> N seed=578, n_estimators=10, ).frame + gap_filled = _late_primary_entry(gap_filled) donor = pd.DataFrame( { "employment_income": [45_000.0, 8_000.0, 70_000.0, 22_000.0], @@ -3166,6 +6043,7 @@ def test_run_stacked_puf_pass_applies_clone_two_capital_gains_tail() -> None: mass_log=gap_filled.mass_log, metadata=gap_filled.metadata, ) + gap_filled = _late_primary_entry(gap_filled) donor = pd.DataFrame( { "tax_unit_id": [10, 20, 1_000_001], @@ -3247,6 +6125,7 @@ def test_run_stacked_puf_pass_applies_clone_two_capital_gains_tail() -> None: assert tail["clone"]["support_role"] == "puf_tax_detail" assert tail["clone"]["source_channels"] == live_source_channels assert "support_channel" not in tail["clone"] + assert tail["late_overlap_ownership"] == dict(us_late_overlap_ownership_receipt()) preservation = stacked_spine_module.assert_stacked_tail_cells_preserved( result.frame, @@ -3254,6 +6133,55 @@ def test_run_stacked_puf_pass_applies_clone_two_capital_gains_tail() -> None: ) assert preservation["passed"] is True assert preservation["tail_owned_cell_count"] == 14 + assert ( + preservation["overlap_ownership_sha256"] + == tail["late_overlap_ownership"]["sha256"] + ) + + forged_ownership = deepcopy(tail) + forged_receipt = forged_ownership["late_overlap_ownership"] + forged_receipt["ownership"][0]["final_owner"] = "forged_owner" + receipt_payload = dict(forged_receipt) + receipt_payload.pop("sha256") + forged_receipt["sha256"] = stacked_spine_module._canonical_sha256(receipt_payload) + manifest_payload = dict(forged_ownership) + manifest_payload.pop("manifest_sha256") + forged_ownership["manifest_sha256"] = stacked_spine_module._canonical_sha256( + manifest_payload + ) + with pytest.raises(ValueError, match="overlap ownership"): + stacked_spine_module.assert_stacked_tail_cells_preserved( + result.frame, + forged_ownership, + ) + + terminal_gates = ( + stacked_completeness_gate(result.frame, tail_manifest=tail), + by_origin_battery(result.frame, tail_manifest=tail), + ) + for gate in terminal_gates: + terminal_support = gate.details["puf_capital_gains_tail_support"] + assert terminal_support["tail_manifest_sha256"] == tail["manifest_sha256"] + assert terminal_support["recipient_support"] == tail["recipient_support"] + + tampered_details = deepcopy(terminal_gates[0].details) + tampered_terminal = tampered_details["puf_capital_gains_tail_support"] + tampered_support = tampered_terminal["recipient_support"] + tampered_support["strata"][0]["observed_count"] += 1 + support_payload = dict(tampered_support) + support_payload.pop("sha256") + tampered_support["sha256"] = stacked_spine_module._canonical_sha256(support_payload) + terminal_payload = dict(tampered_terminal) + terminal_payload.pop("sha256") + tampered_terminal["sha256"] = stacked_spine_module._canonical_sha256( + terminal_payload + ) + with pytest.raises(ValueError, match="recipient-support candidate count"): + stacked_spine_module._validate_stacked_gate_manifest_details( + terminal_gates[0].name, + tampered_details, + passed=terminal_gates[0].passed, + ) multi_person_record = next( record @@ -3453,7 +6381,7 @@ def overlapping_source(stratum: str, income_shift: float) -> Frame: } ) result = run_stacked_puf_pass( - stacked, + _late_primary_entry(stacked), donor, clone_attachment_fraction=1.0, clone_attachment_seed=578, @@ -4033,7 +6961,7 @@ def test_self_digested_partial_authority_cannot_forge_production_identity() -> N GateReport((result,)).to_manifest() -@pytest.mark.parametrize("stale_version", (1, 2, 3, 4, 5)) +@pytest.mark.parametrize("stale_version", (1, 2, 3, 4, 5, 6, 7, 8, 9)) def test_self_consistent_stale_stacked_authority_versions_are_rejected( stale_version: int, ) -> None: @@ -4053,7 +6981,7 @@ def test_self_consistent_stale_stacked_authority_versions_are_rejected( ) stale_receipt = stacked_spine_module._authority_receipt(stale) - assert stacked_spine_module.stacked_spine_authority_receipt()["version"] == 6 + assert stacked_spine_module.stacked_spine_authority_receipt()["version"] == 10 assert stale_receipt["version"] == stale_version assert stale_receipt["integrity_valid"] is True assert stale_receipt["digest_matches_declared"] is True @@ -4068,6 +6996,45 @@ def test_self_consistent_stale_stacked_authority_versions_are_rejected( ) +def test_stacked_authority_binds_import_validated_late_producer_schedule() -> None: + receipt = stacked_spine_module.stacked_spine_authority_receipt() + component = receipt["components"]["late_producer_schedule"] + + assert receipt["version"] == 10 + assert component["producer_count"] == 38 + assert component["schedule_sha256"] == ( + stacked_spine_module.CANONICAL_US_LATE_PRODUCER_SCHEDULE.sha256 + ) + assert component["identity"]["status"] == "derived_and_import_validated" + assert component["digest_matches_declared"] is True + + +def test_rebound_late_producer_schedule_invalidates_production_authority( + monkeypatch: pytest.MonkeyPatch, +) -> None: + live = dict(stacked_spine_module.us_late_producer_schedule_receipt()) + live["schedule_sha256"] = "0" * 64 + monkeypatch.setattr( + stacked_spine_module, + "us_late_producer_schedule_receipt", + lambda: live, + ) + + authority = stacked_spine_module._production_stacked_authority() + receipt = stacked_spine_module._authority_receipt(authority) + + assert receipt["canonical"] is False + assert ( + receipt["components"]["late_producer_schedule"]["digest_matches_declared"] + is False + ) + with pytest.raises(ValueError, match="non-canonical stacked authority"): + stacked_spine_module._validate_production_authority_receipt( + receipt, + boundary="rebound late producer schedule", + ) + + def test_rebound_anchor_aliases_cannot_replace_captured_canonical_authority( monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -4466,7 +7433,9 @@ def test_stripped_noncanonical_receipt_cannot_escape_under_a_renamed_gate( GateReport((stripped,)).to_manifest() -def test_stripped_six_component_authority_cannot_escape_under_a_renamed_gate() -> None: +def test_stripped_eight_component_authority_cannot_escape_under_a_renamed_gate() -> ( + None +): authority = stacked_spine_module.stacked_spine_authority_receipt() components = deepcopy(dict(authority["components"])) assert set(components) == { @@ -4476,6 +7445,8 @@ def test_stripped_six_component_authority_cannot_escape_under_a_renamed_gate() - "metric_registry", "joint_metric_registry", "support_profile", + "puf_capital_gains_tail_support_contract", + "late_producer_schedule", } stripped = GateResult( name="renamed_stacked_battery", @@ -4662,8 +7633,8 @@ def with_columns(frame: Frame, position: int) -> Frame: ).frame -@pytest.mark.parametrize("clone_role", (0, 1, 2)) -def test_completeness_rejects_nonfinite_values_on_every_clone_role( +@pytest.mark.parametrize("clone_role", (0, 1)) +def test_completeness_rejects_nonfinite_values_on_every_non_tail_clone_role( clone_role: int, ) -> None: base = _battery_frame( @@ -4680,10 +7651,6 @@ def test_completeness_rejects_nonfinite_values_on_every_clone_role( clone_attachment_seed=578, ) tables = {entity: attached.table(entity).copy() for entity in attached.entities} - if clone_role == 2: - for entity, table in tables.items(): - clone_column = support_clone_index_column(entity) - table.loc[table[clone_column].eq(1), clone_column] = 2 person = tables["person"] clone_column = support_clone_index_column("person") invalid = person[clone_column].eq(clone_role) @@ -4718,6 +7685,36 @@ def test_completeness_rejects_nonfinite_values_on_every_clone_role( json.dumps(GateReport((result,)).to_manifest(), allow_nan=False) +def test_terminal_gate_requires_manifest_for_live_clone_two_rows() -> None: + attached = clone_us_frame_for_puf_support( + _battery_frame( + { + "taxable_interest_income": ( + np.asarray([100.0] * 8), + np.asarray([100.0] * 11), + ) + } + ), + clone_attachment_fraction=1.0, + clone_attachment_seed=578, + ) + tables = {entity: attached.table(entity).copy() for entity in attached.entities} + for entity, table in tables.items(): + clone_column = support_clone_index_column(entity) + table.loc[table[clone_column].eq(1), clone_column] = 2 + frame = Frame( + tables, + attached.schema, + {entity: attached.weights_for(entity) for entity in attached.weighted_entities}, + attached.strata, + mass_log=attached.mass_log, + metadata=attached.metadata, + ) + + with pytest.raises(ValueError, match="clone-2 rows require the bound"): + stacked_completeness_gate(frame) + + @pytest.mark.parametrize( ("asec_values", "acs_values", "expected_invalid"), ( @@ -5308,7 +8305,7 @@ def test_end_to_end_stack_gap_fill_puf_pass_gates_and_battery(tmp_path) -> None: } ) passed = stacked_spine_module._run_stacked_puf_pass_without_tail_for_test( - gap_filled.frame, + _late_primary_entry(gap_filled.frame), donor, clone_attachment_fraction=0.5, clone_attachment_seed=578, @@ -5326,7 +8323,7 @@ def test_end_to_end_stack_gap_fill_puf_pass_gates_and_battery(tmp_path) -> None: # zero-fill: without gap-fill the strict doctrine refuses the PUF pass. with pytest.raises(ValueError, match="puf_predictor_taxable_interest_income"): stacked_spine_module._run_stacked_puf_pass_without_tail_for_test( - stacked, + _late_primary_entry(stacked), donor, clone_attachment_fraction=0.5, clone_attachment_seed=578, diff --git a/packages/microcosm-frame/src/microcosm/frame/__init__.py b/packages/microcosm-frame/src/microcosm/frame/__init__.py index 3dd344b4..c0a97a49 100644 --- a/packages/microcosm-frame/src/microcosm/frame/__init__.py +++ b/packages/microcosm-frame/src/microcosm/frame/__init__.py @@ -16,7 +16,14 @@ wsum, ) from microcosm.frame.bundle import CONSERVE_MASS, DEFAULT_STRATUM, Frame -from microcosm.frame.materialize import engine_tables +from microcosm.frame.materialize import ( + PyTablesBooleanMaterialization, + engine_tables, + materialize_nullable_booleans_for_pytables, + nullable_boolean_values_and_mask, + put_frame_table, + read_frame_table, +) from microcosm.frame.rules import ExportContract, RulesEngine from microcosm.frame.schema import EntitySchema, LinkSpec, VariableMetadata from microcosm.frame.units import ( @@ -49,6 +56,7 @@ "LinkSpec", "MassChange", "MassChangeRecord", + "PyTablesBooleanMaterialization", "RulesEngine", "VariableMetadata", "WeightKind", @@ -58,6 +66,10 @@ "engine_tables", "gini", "groupby_wsum", + "materialize_nullable_booleans_for_pytables", + "nullable_boolean_values_and_mask", + "put_frame_table", + "read_frame_table", "wmean", "wmedian", "wquantile", diff --git a/packages/microcosm-frame/src/microcosm/frame/adapters/axiom.py b/packages/microcosm-frame/src/microcosm/frame/adapters/axiom.py index a9875f7b..623639f8 100644 --- a/packages/microcosm-frame/src/microcosm/frame/adapters/axiom.py +++ b/packages/microcosm-frame/src/microcosm/frame/adapters/axiom.py @@ -58,7 +58,7 @@ import pandas as pd from microcosm.frame.bundle import Frame -from microcosm.frame.materialize import engine_tables +from microcosm.frame.materialize import engine_tables, put_frame_table, read_frame_table from microcosm.frame.rules import ExportContract from microcosm.frame.schema import EntitySchema, VariableMetadata @@ -135,9 +135,11 @@ def __init__( self._schema = schema self._contract = contract if contract is not None else ExportContract.empty() self._defaults = dict(defaults or {}) - self._entity_names = dict(entity_names) if entity_names is not None else { - entity: entity.capitalize() for entity in schema.entities - } + self._entity_names = ( + dict(entity_names) + if entity_names is not None + else {entity: entity.capitalize() for entity in schema.entities} + ) unknown = sorted(set(self._entity_names) - set(schema.entities)) if unknown: raise ValueError( @@ -441,9 +443,7 @@ def _program(self, frame_entity: str, *, missing_ok: bool = False) -> Any: ) from None self._programs[frame_entity] = program if self._metadata is None: - self._metadata = { - item.name: item for item in program.derived_metadata - } + self._metadata = {item.name: item for item in program.derived_metadata} return program def _derived_metadata(self) -> dict[str, Any]: @@ -579,8 +579,7 @@ def __init__( return if tables is None or time_period is None: raise ValueError( - "AxiomEntityTableDataset needs tables and time_period (or " - "file_path)." + "AxiomEntityTableDataset needs tables and time_period (or file_path)." ) self.tables = {name: table.copy() for name, table in tables.items()} self.time_period = int(time_period) @@ -614,7 +613,13 @@ def save(self, file_path: str | Path) -> None: with pd.HDFStore(str(path)) as store: for name, table in self.tables.items(): if len(table) > 0: - store.put(name, table, format="table", data_columns=True) + put_frame_table( + store, + name, + table, + preferred_format="table", + data_columns=True, + ) store.put( self._TIME_PERIOD_KEY, pd.Series([int(self.time_period)]), @@ -633,7 +638,7 @@ def _read(cls, path: Path) -> tuple[dict[str, pd.DataFrame], int]: if name == cls._TIME_PERIOD_KEY: time_period = int(store[key].iloc[0]) continue - tables[name] = store[key] + tables[name] = read_frame_table(store, key) if time_period is None: raise ValueError(f"Dataset at {path} carries no {cls._TIME_PERIOD_KEY}.") return tables, time_period @@ -649,12 +654,7 @@ def _period_bounds(period: int | str) -> tuple[str, str, str]: text = str(period) if len(text) == 4 and text.isdigit(): return f"{text}-01-01", f"{text}-12-31", "calendar_year" - if ( - len(text) == 7 - and text[4] == "-" - and text[:4].isdigit() - and text[5:].isdigit() - ): + if len(text) == 7 and text[4] == "-" and text[:4].isdigit() and text[5:].isdigit(): year, month = int(text[:4]), int(text[5:]) if not 1 <= month <= 12: raise ValueError(f"Invalid month in period {period!r}.") diff --git a/packages/microcosm-frame/src/microcosm/frame/adapters/policyengine_us.py b/packages/microcosm-frame/src/microcosm/frame/adapters/policyengine_us.py index 0e2c3adc..236b6d7e 100644 --- a/packages/microcosm-frame/src/microcosm/frame/adapters/policyengine_us.py +++ b/packages/microcosm-frame/src/microcosm/frame/adapters/policyengine_us.py @@ -17,7 +17,9 @@ column, materialized from the frame's typed household weights. """ +import json from collections.abc import Iterable, Mapping, Sequence +from dataclasses import dataclass from functools import lru_cache from hashlib import sha256 from importlib.metadata import PackageNotFoundError, distribution @@ -37,7 +39,12 @@ _index_policyengine_us_sources as _build_policyengine_us_source_index, ) from microcosm.frame.bundle import Frame -from microcosm.frame.materialize import engine_tables +from microcosm.frame.materialize import ( + engine_tables, + materialize_nullable_booleans_for_pytables, + put_frame_table, + read_frame_table, +) from microcosm.frame.rules import ExportContract from microcosm.frame.schema import EntitySchema, VariableMetadata from microcosm.frame.units import US_SCHEMA @@ -46,6 +53,7 @@ "ConsumerReceipt", "PolicyEngineUSEngine", "PolicyEngineUSVariableMetadataIndex", + "VariableDependencyClosure", ] _PERSON_TABLE = "person" @@ -74,6 +82,26 @@ } ) + +@dataclass(frozen=True) +class VariableDependencyClosure: + """Static transitive variable graph for one PolicyEngine output. + + Edges are ordered ``(consumer, dependency)`` pairs and are deduplicated + across source-reference sites. The digest binds the installed engine + version and the complete normalized graph, so a checked-in downstream + input manifest can fail closed on either source or dependency drift + without executing a microsimulation. + """ + + engine_version: str + root: str + input_leaves: tuple[str, ...] + formula_nodes: tuple[str, ...] + edges: tuple[tuple[str, str], ...] + sha256: str + + # PolicyEngine ``value_type`` (a Python type) → kernel dtype kind. Enum value # types are not listed and fall back to ``"str"`` at the call site. _DTYPE_KIND_BY_VALUE_TYPE: dict[type, str] = { @@ -273,6 +301,7 @@ def __init__(self) -> None: source_index = _installed_policyengine_us_variable_sources() self._definitions = source_index.definitions self._consumers = source_index.consumers + self._engine_version = distribution("policyengine-us").version def variable_metadata(self, name: str) -> VariableMetadata: definition = self._definitions.get(name) @@ -295,6 +324,66 @@ def consumer_receipts(self, name: str) -> tuple[ConsumerReceipt, ...]: raise ValueError(f"Unknown PolicyEngine-US source variable {name!r}.") return self._consumers.get(name, ()) + def variable_dependency_closure(self, name: str) -> VariableDependencyClosure: + """Return the statically authenticated transitive graph for ``name``. + + The source index records references in the target-to-consumer + direction. This method inverts those receipts, walks outward from the + requested output, and classifies each reachable definition exactly as + :meth:`variables` does. Multiple source sites for the same reference + collapse to one semantic edge. + """ + + if name not in self._definitions: + raise ValueError(f"Unknown PolicyEngine-US source variable {name!r}.") + + dependencies: dict[str, set[str]] = {} + for target, receipts in self._consumers.items(): + for receipt in receipts: + if receipt.consumer in self._definitions: + dependencies.setdefault(receipt.consumer, set()).add(target) + + reachable: set[str] = set() + edges: set[tuple[str, str]] = set() + pending = [name] + while pending: + consumer = pending.pop() + if consumer in reachable: + continue + reachable.add(consumer) + for target in dependencies.get(consumer, ()): + edges.add((consumer, target)) + if target not in reachable: + pending.append(target) + + inputs = set(self.variables()) + input_leaves = tuple(sorted(reachable & inputs)) + formula_nodes = tuple(sorted(reachable - inputs)) + ordered_edges = tuple(sorted(edges)) + payload = { + "engine_version": self._engine_version, + "root": name, + "input_leaves": list(input_leaves), + "formula_nodes": list(formula_nodes), + "edges": [list(edge) for edge in ordered_edges], + } + digest = sha256( + json.dumps( + payload, + allow_nan=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + ).hexdigest() + return VariableDependencyClosure( + engine_version=self._engine_version, + root=name, + input_leaves=input_leaves, + formula_nodes=formula_nodes, + edges=ordered_edges, + sha256=digest, + ) + def formula_owned_outputs(self, names: Iterable[str]) -> set[str]: requested = set(names) return set(requested & _FORMULA_OWNED_COMPAT_COLUMNS) | { @@ -912,10 +1001,12 @@ def _write_and_verify( period: int, output_path: Path, ) -> None: - """Persist tables as a ``USSingleYearDataset`` and verify the round-trip. + """Persist PolicyEngine-US tables and verify its dataset round-trip. - Saves the dataset, reloads it, and asserts every column from a - non-empty table survived (``.save`` only writes tables with rows). + This owns the same entity-table HDF layout as ``USSingleYearDataset`` + while routing every Frame table through Microcosm's nullable-boolean + boundary. It then reloads through ``USSingleYearDataset`` and asserts + every column from a non-empty table survived. Raises: ValueError: If a column expected after reload is missing. @@ -923,8 +1014,27 @@ def _write_and_verify( from policyengine_us.data import USSingleYearDataset output_path.parent.mkdir(parents=True, exist_ok=True) - dataset = self._build_dataset(tables, period) - dataset.save(str(output_path)) + output_path.unlink(missing_ok=True) + materialized_tables = { + name: materialize_nullable_booleans_for_pytables(table).table + for name, table in tables.items() + } + with pd.HDFStore(str(output_path), mode="w") as store: + for name in (_PERSON_TABLE, *_GROUP_TABLES): + table = tables[name] + if len(table) > 0: + put_frame_table( + store, + name, + table, + preferred_format="table", + data_columns=True, + ) + store.put( + "_time_period", + pd.Series([int(period)]), + format="table", + ) expected_columns: set[str] = set() for frame in tables.values(): @@ -932,12 +1042,19 @@ def _write_and_verify( expected_columns.update(frame.columns) reloaded = USSingleYearDataset(file_path=str(output_path)) + with pd.HDFStore(str(output_path), mode="r") as store: + logical_tables = { + name: read_frame_table(store, name) + for name in (_PERSON_TABLE, *_GROUP_TABLES) + if len(tables[name]) > 0 + } persisted_columns: set[str] = set() dtype_mismatches: list[str] = [] for name in (_PERSON_TABLE, *_GROUP_TABLES): - reloaded_table = getattr(reloaded, name) - persisted_columns.update(reloaded_table.columns) - source_table = tables.get(name) + external_table = getattr(reloaded, name) + reloaded_table = logical_tables.get(name, external_table) + persisted_columns.update(external_table.columns) + source_table = materialized_tables.get(name) if source_table is None or len(source_table) == 0: continue for column in source_table.columns: diff --git a/packages/microcosm-frame/src/microcosm/frame/materialize.py b/packages/microcosm-frame/src/microcosm/frame/materialize.py index b03cd218..be49c5eb 100644 --- a/packages/microcosm-frame/src/microcosm/frame/materialize.py +++ b/packages/microcosm-frame/src/microcosm/frame/materialize.py @@ -16,15 +16,248 @@ from __future__ import annotations +import json +import warnings from collections.abc import Iterable +from dataclasses import dataclass +from typing import Any +import numpy as np import pandas as pd from microcosm.frame.bundle import Frame -__all__ = ["engine_tables"] +__all__ = [ + "PyTablesBooleanMaterialization", + "engine_tables", + "materialize_nullable_booleans_for_pytables", + "nullable_boolean_values_and_mask", + "put_frame_table", + "read_frame_table", +] _WEIGHT_COLUMN_SUFFIX = "_weight" +_PYTABLES_FRAME_TABLE_CODEC_ATTR = "_microcosm_frame_table_codec" +_PYTABLES_FRAME_TABLE_CODEC_VERSION = 1 + + +@dataclass(frozen=True) +class PyTablesBooleanMaterialization: + """A table made safe for pandas' two PyTables storage formats. + + ``nullable_columns`` records every pandas ``BooleanDtype`` input; + ``missing_columns`` is the subset represented as object-backed Python + booleans plus ``pd.NA``. A table with any such column requires fixed HDF + format because PyTables table format has no nullable-boolean column type. + """ + + table: pd.DataFrame + nullable_columns: tuple[str, ...] + missing_columns: tuple[str, ...] + + def hdf_format(self, preferred: str) -> str: + """Return ``preferred`` unless missing booleans require fixed format.""" + + if preferred not in {"fixed", "table"}: + raise ValueError("preferred HDF format must be 'fixed' or 'table'.") + return "fixed" if self.missing_columns else preferred + + +def nullable_boolean_values_and_mask( + series: pd.Series, +) -> tuple[np.ndarray, np.ndarray]: + """Return canonical bool values and the exact NA mask for BooleanDtype. + + Missing positions always carry a false value bit. This makes the value + bytes deterministic and prevents data hidden underneath the mask from + changing serialized artifacts. + """ + + if not isinstance(series, pd.Series): + raise TypeError(f"series must be a pandas Series, got {type(series).__name__}.") + if not isinstance(series.dtype, pd.BooleanDtype): + raise TypeError( + "nullable_boolean_values_and_mask requires pandas BooleanDtype, " + f"got {series.dtype!s}." + ) + values = series.to_numpy( + dtype=np.bool_, + na_value=False, + copy=False, + ) + mask = series.isna().to_numpy(dtype=np.bool_, copy=False) + if values.ndim != 1 or mask.ndim != 1 or len(values) != len(mask): + raise RuntimeError("Nullable-boolean materialization produced invalid shape.") + if values[mask].any(): # pragma: no cover - defensive canonicality assertion + raise RuntimeError("Nullable-boolean null positions must carry false bits.") + return values, mask + + +def materialize_nullable_booleans_for_pytables( + table: pd.DataFrame, +) -> PyTablesBooleanMaterialization: + """Materialize pandas nullable booleans without changing their semantics. + + Complete columns become native NumPy bool with byte-identical logical + values. Columns containing NA become object-backed Python ``bool`` plus + the explicit ``pd.NA`` sentinel; pandas fixed-format HDF preserves that + representation losslessly. The source table is never mutated, and a + shallow boundary copy is allocated only when a nullable boolean exists. + """ + + if not isinstance(table, pd.DataFrame): + raise TypeError( + f"table must be a pandas DataFrame, got {type(table).__name__}." + ) + result = table + nullable_columns: list[str] = [] + missing_columns: list[str] = [] + for column in table.columns: + series = table[column] + if not isinstance(series.dtype, pd.BooleanDtype): + continue + if not isinstance(column, str): + raise TypeError("Nullable-boolean table columns must have string names.") + if result is table: + result = table.copy(deep=False) + nullable_columns.append(column) + values, mask = nullable_boolean_values_and_mask(series) + if mask.any(): + object_values = values.astype(object) + object_values[mask] = pd.NA + result[column] = pd.Series( + object_values, + index=series.index, + name=series.name, + dtype=object, + copy=False, + ) + missing_columns.append(column) + else: + result[column] = pd.Series( + values, + index=series.index, + name=series.name, + dtype=np.bool_, + copy=False, + ) + return PyTablesBooleanMaterialization( + table=result, + nullable_columns=tuple(nullable_columns), + missing_columns=tuple(missing_columns), + ) + + +def put_frame_table( + store: Any, + key: str, + table: pd.DataFrame, + *, + preferred_format: str, + data_columns: bool | list[str] | None = None, +) -> PyTablesBooleanMaterialization: + """Write one Frame table through the shared nullable-boolean boundary.""" + + materialized = materialize_nullable_booleans_for_pytables(table) + hdf_format = materialized.hdf_format(preferred_format) + options: dict[str, object] = {} + if hdf_format == "table" and data_columns is not None: + options["data_columns"] = data_columns + with warnings.catch_warnings(): + warnings.simplefilter("ignore", pd.errors.PerformanceWarning) + store.put( + key, + materialized.table, + format=hdf_format, + **options, + ) + if materialized.missing_columns: + codec = { + "codec_version": _PYTABLES_FRAME_TABLE_CODEC_VERSION, + "nullable_boolean_columns": list(materialized.missing_columns), + } + setattr( + store.get_storer(key).attrs, + _PYTABLES_FRAME_TABLE_CODEC_ATTR, + json.dumps(codec, sort_keys=True, separators=(",", ":")), + ) + return materialized + + +def read_frame_table(store: Any, key: str) -> pd.DataFrame: + """Read one Frame table and restore its versioned PyTables dtypes. + + The table-local codec attribute is absent from older artifacts and from + tables that needed no explicit nullable representation. When present it + identifies object-backed BooleanDtype columns whose exact null positions + PyTables preserves but which pandas may otherwise infer as strings when + every value is missing. + """ + + table = store[key] + raw_codec = getattr( + store.get_storer(key).attrs, + _PYTABLES_FRAME_TABLE_CODEC_ATTR, + None, + ) + if raw_codec is None: + return table + try: + codec = json.loads(raw_codec) + except (TypeError, ValueError) as exc: + raise ValueError( + f"Frame table {key!r} has malformed dtype codec metadata." + ) from exc + expected_fields = {"codec_version", "nullable_boolean_columns"} + if not isinstance(codec, dict) or set(codec) != expected_fields: + raise ValueError( + f"Frame table {key!r} dtype codec must contain exactly " + f"{sorted(expected_fields)!r}." + ) + if type(codec["codec_version"]) is not int or ( + codec["codec_version"] != _PYTABLES_FRAME_TABLE_CODEC_VERSION + ): + raise ValueError( + f"Frame table {key!r} has unsupported dtype codec version " + f"{codec['codec_version']!r}." + ) + columns = codec["nullable_boolean_columns"] + if ( + not isinstance(columns, list) + or not columns + or any(not isinstance(column, str) or not column for column in columns) + or len(columns) != len(set(columns)) + ): + raise ValueError( + f"Frame table {key!r} has invalid nullable-boolean column metadata." + ) + missing_columns = sorted(set(columns) - set(table.columns)) + if missing_columns: + raise ValueError( + f"Frame table {key!r} dtype codec names absent column(s): " + f"{missing_columns!r}." + ) + + restored = table.copy(deep=False) + for column in columns: + series = table[column] + invalid = series.notna() & ~series.map(lambda value: isinstance(value, bool)) + if invalid.any(): + raise ValueError( + f"Frame table {key!r} nullable-boolean column {column!r} " + "contains a non-boolean value." + ) + values = series.to_numpy(dtype=object, copy=True) + null_mask = series.isna().to_numpy(dtype=np.bool_, copy=False) + values[null_mask] = pd.NA + restored[column] = pd.Series( + values, + index=series.index, + name=series.name, + dtype=object, + copy=False, + ) + return restored def engine_tables( diff --git a/packages/microcosm-frame/tests/test_materialize.py b/packages/microcosm-frame/tests/test_materialize.py index 307bf64c..7f058b0f 100644 --- a/packages/microcosm-frame/tests/test_materialize.py +++ b/packages/microcosm-frame/tests/test_materialize.py @@ -11,7 +11,17 @@ import pandas as pd import pytest -from microcosm.frame import EntitySchema, Frame, WeightKind, Weights, engine_tables +from microcosm.frame import ( + EntitySchema, + Frame, + WeightKind, + Weights, + engine_tables, + materialize_nullable_booleans_for_pytables, + nullable_boolean_values_and_mask, + put_frame_table, + read_frame_table, +) def _uk_frame(*, stale_weight_column: bool) -> Frame: @@ -104,3 +114,127 @@ def test_simple_schema_bundle_matches_typed_weights(make_bundle) -> None: tables["household"]["household_weight"].to_numpy(), bundle.weights_for("household").values, ) + + +def test_nullable_boolean_values_are_canonical_under_the_null_mask() -> None: + series = pd.Series([True, pd.NA, False], dtype="boolean") + + values, mask = nullable_boolean_values_and_mask(series) + + assert values.dtype == np.dtype(np.bool_) + assert ( + values.tobytes() == np.asarray([True, False, False], dtype=np.bool_).tobytes() + ) + np.testing.assert_array_equal(mask, np.asarray([False, True, False])) + + +def test_pytables_boolean_materialization_is_lossless_and_non_mutating() -> None: + source = pd.DataFrame( + { + "native": np.asarray([False, True, False], dtype=np.bool_), + "complete": pd.Series([True, False, True], dtype="boolean"), + "missing": pd.Series([True, pd.NA, False], dtype="boolean"), + } + ) + before = source.copy(deep=True) + + materialized = materialize_nullable_booleans_for_pytables(source) + + pd.testing.assert_frame_equal(source, before, check_exact=True, check_dtype=True) + assert materialized.table is not source + assert materialized.nullable_columns == ("complete", "missing") + assert materialized.missing_columns == ("missing",) + assert materialized.table["native"].dtype == np.dtype(np.bool_) + assert materialized.table["complete"].dtype == np.dtype(np.bool_) + assert materialized.table["complete"].to_numpy(copy=False).tobytes() == ( + source["complete"].to_numpy(dtype=np.bool_, copy=False).tobytes() + ) + assert materialized.table["missing"].dtype == np.dtype(object) + assert materialized.table["missing"].tolist() == [True, pd.NA, False] + assert materialized.hdf_format("table") == "fixed" + assert materialized.hdf_format("fixed") == "fixed" + + +def test_pytables_writer_uses_native_bool_or_fixed_explicit_na(tmp_path) -> None: + pytest.importorskip("tables") + path = tmp_path / "nullable-booleans.h5" + complete = pd.DataFrame({"flag": pd.Series([True, False, True], dtype="boolean")}) + missing = pd.DataFrame({"flag": pd.Series([True, pd.NA, False], dtype="boolean")}) + + with pd.HDFStore(path, mode="w") as store: + put_frame_table( + store, + "complete", + complete, + preferred_format="table", + data_columns=True, + ) + put_frame_table( + store, + "missing", + missing, + preferred_format="table", + data_columns=True, + ) + assert store.get_storer("complete").is_table is True + assert store.get_storer("missing").is_table is False + + with pd.HDFStore(path, mode="r") as store: + stored_complete = store["complete"]["flag"] + stored_missing = store["missing"]["flag"] + assert stored_complete.dtype == np.dtype(np.bool_) + assert ( + stored_complete.to_numpy(copy=False).tobytes() + == np.asarray([True, False, True], dtype=np.bool_).tobytes() + ) + assert stored_missing.dtype == np.dtype(object) + assert stored_missing.tolist() == [True, pd.NA, False] + + +def test_pytables_reader_restores_all_missing_nullable_boolean(tmp_path) -> None: + pytest.importorskip("tables") + path = tmp_path / "all-missing-nullable-boolean.h5" + source = pd.DataFrame({"flag": pd.Series([pd.NA, pd.NA, pd.NA], dtype="boolean")}) + + with pd.HDFStore(path, mode="w") as store: + put_frame_table(store, "person", source, preferred_format="fixed") + + with pd.HDFStore(path, mode="r") as store: + raw = store["person"] + restored = read_frame_table(store, "person") + + assert raw["flag"].dtype.kind in {"O", "U"} + assert restored["flag"].dtype == np.dtype(object) + assert restored["flag"].tolist() == [pd.NA, pd.NA, pd.NA] + pd.testing.assert_series_equal( + restored["flag"].astype("boolean"), + source["flag"], + ) + + +@pytest.mark.parametrize( + ("payload", "match"), + [ + ("not-json", "malformed dtype codec"), + ( + '{"codec_version":2,"nullable_boolean_columns":["flag"]}', + "unsupported dtype codec version", + ), + ( + '{"codec_version":1,"nullable_boolean_columns":["absent"]}', + "names absent column", + ), + ], +) +def test_pytables_reader_rejects_invalid_dtype_codec( + tmp_path, payload: str, match: str +) -> None: + pytest.importorskip("tables") + path = tmp_path / "invalid-codec.h5" + with pd.HDFStore(path, mode="w") as store: + store.put("person", pd.DataFrame({"flag": [True]}), format="fixed") + store.get_storer("person").attrs._microcosm_frame_table_codec = payload + + with pd.HDFStore(path, mode="r") as store: + with pytest.raises(ValueError, match=match): + read_frame_table(store, "person") diff --git a/supabase/migrations/20260813000000_logbook_f004_rung.sql b/supabase/migrations/20260813000000_logbook_f004_rung.sql new file mode 100644 index 00000000..dc0b0c57 --- /dev/null +++ b/supabase/migrations/20260813000000_logbook_f004_rung.sql @@ -0,0 +1,9 @@ +-- #624 ladder revision: add the 4% smoke rung token to the builds rung +-- constraint. Rung grammar history: f001/f010/f100 at genesis; f004 added +-- when measurement showed 1% sits below the tail's own support floor while +-- 4% clears every filing-status stratum naturally. +ALTER TABLE logbook.builds + DROP CONSTRAINT builds_rung_fraction_token; +ALTER TABLE logbook.builds + ADD CONSTRAINT builds_rung_fraction_token + CHECK (rung IN ('f001', 'f004', 'f010', 'f100')); diff --git a/tools/_legacy/build_us_acs_multispine_base.py b/tools/_legacy/build_us_acs_multispine_base.py index 09d9d98c..f6b8e754 100644 --- a/tools/_legacy/build_us_acs_multispine_base.py +++ b/tools/_legacy/build_us_acs_multispine_base.py @@ -15,7 +15,6 @@ import hashlib import json import os -import warnings from dataclasses import replace from pathlib import Path from typing import Any @@ -53,7 +52,13 @@ UsPumaLadder, load_us_puma_ladder, ) -from microcosm.frame import Frame, WeightKind, Weights +from microcosm.frame import ( + Frame, + WeightKind, + Weights, + put_frame_table, + read_frame_table, +) from microcosm.frame.units import US_SCHEMA PERIOD = 2024 @@ -965,17 +970,11 @@ def _spine_totals(frame: Frame) -> dict[str, dict[str, Any]]: def _load_base_frame(path: Path) -> Frame: """Load the dense donor H5 without importing PolicyEngine-US at tool import.""" - from policyengine_us.data import USSingleYearDataset - - dataset = USSingleYearDataset(file_path=str(path)) - tables = { - "person": dataset.person, - "household": dataset.household, - "tax_unit": dataset.tax_unit, - "spm_unit": dataset.spm_unit, - "family": dataset.family, - "marital_unit": dataset.marital_unit, - } + with pd.HDFStore(path, mode="r") as store: + tables = { + entity: read_frame_table(store, entity) for entity in US_SCHEMA.entities + } + tables["household"] = tables["household"].copy() household_weights = ( tables["household"].pop("household_weight").to_numpy(dtype=np.float64) ) @@ -1010,12 +1009,12 @@ def _write_dataset( table = table.copy() table["household_weight"] = frame.weights_for("household").values if len(table): - # Fixed format preserves mixed bool/null object columns - # losslessly. Table format rejects them, which would force - # an unauthorized fill or type rewrite on base-only inputs. - with warnings.catch_warnings(): - warnings.simplefilter("ignore", pd.errors.PerformanceWarning) - store.put(entity, table, format="fixed") + put_frame_table( + store, + entity, + table, + preferred_format="fixed", + ) store.put( "_time_period", pd.Series([int(period)]), @@ -1045,7 +1044,7 @@ def _write_dataset( expected = frame.table(entity) if not len(expected): continue - stored = store[entity] + stored = read_frame_table(store, entity) expected_columns = list(expected.columns) if entity == "household": expected_columns.append("household_weight") diff --git a/tools/build_us_acs_local_release.py b/tools/build_us_acs_local_release.py index 6989285b..f597cf8c 100644 --- a/tools/build_us_acs_local_release.py +++ b/tools/build_us_acs_local_release.py @@ -54,7 +54,6 @@ import sys import threading import time -import warnings from datetime import UTC, datetime from pathlib import Path @@ -629,6 +628,8 @@ def write_lean_checkpoint( ): """Assemble the lean target-frame H5 + targets.json (memory-bounded).""" + from microcosm.frame import put_frame_table + checkpoint_dir.mkdir(parents=True, exist_ok=True) if isinstance(admin_matrix, np.memmap): admin_matrix = np.array(admin_matrix) @@ -646,13 +647,26 @@ def write_lean_checkpoint( lean_households["household_weight"] = struct["weights"] checkpoint_h5 = checkpoint_dir / "target_frame_lean.h5" with pd.HDFStore(checkpoint_h5, mode="w") as store: - with warnings.catch_warnings(): - warnings.simplefilter("ignore", pd.errors.PerformanceWarning) - store.put("household", lean_households, format="fixed") - store.put("person", struct["person"], format="fixed") - for group, table in struct["groups"].items(): - store.put(group, table, format="fixed") - store.put("_time_period", pd.Series([PERIOD]), format="table") + put_frame_table( + store, + "household", + lean_households, + preferred_format="fixed", + ) + put_frame_table( + store, + "person", + struct["person"], + preferred_format="fixed", + ) + for group, table in struct["groups"].items(): + put_frame_table( + store, + group, + table, + preferred_format="fixed", + ) + store.put("_time_period", pd.Series([PERIOD]), format="table") targets = [ dict( name=target["name"], @@ -685,13 +699,13 @@ def write_lean_checkpoint( def load_lean_frame(checkpoint_h5: Path): - from microcosm.frame import Frame, WeightKind, Weights + from microcosm.frame import Frame, WeightKind, Weights, read_frame_table from microcosm.frame.units import US_SCHEMA tables = {} with pd.HDFStore(checkpoint_h5, mode="r") as store: for key in ["household", "person"] + list(GROUP_IDS): - tables[key] = store[key] + tables[key] = read_frame_table(store, key) design_weights = tables["household"].pop("household_weight").to_numpy(np.float64) return ( Frame( diff --git a/tools/build_us_fiscal_refresh_release.py b/tools/build_us_fiscal_refresh_release.py index 3681975a..1a1f73cb 100644 --- a/tools/build_us_fiscal_refresh_release.py +++ b/tools/build_us_fiscal_refresh_release.py @@ -391,7 +391,7 @@ class PoolReleaseIdentityMismatchError(ValueError): # so pre-#557 release-refitted surfaces cannot mix with preserved surfaces. "target_frame_materializer_identity_sha256", ) -TARGET_FRAME_CHECKPOINT_SCHEMA_VERSION = 1 +TARGET_FRAME_CHECKPOINT_SCHEMA_VERSION = 2 # 2: the medicaid_take_up stage (microcosm #331) changed base_frame's # takes_up_medicaid_if_eligible before target-frame materialization, so # medicaid_enrolled target columns differ from version-1 checkpoints; the @@ -422,7 +422,10 @@ class PoolReleaseIdentityMismatchError(ValueError): # 10: #557 preserves the staged retirement-distribution surface through # release materialization; pre-#557 checkpoints can carry QRF-refitted leaves # and must not serve the preserved-surface baseline. -TARGET_FRAME_CHECKPOINT_MATERIALIZER_VERSION = 10 +# 11: target-frame checkpoint columns now preserve nullable booleans as +# canonical bool values plus an explicit uint8 null mask. Older checkpoints +# cannot attest this lossless physical representation. +TARGET_FRAME_CHECKPOINT_MATERIALIZER_VERSION = 11 DEFAULT_MAXIMUM_MICROSIM_BATCH_SIZE = 5_000 DEFAULT_L0_REFIT_LAMBDA_SHARE = 0.8 DEFAULT_US_FISCAL_CALIBRATION_EPOCHS = 1_500 @@ -2263,11 +2266,26 @@ def _read_checkpoint_series(group) -> pd.Series: def _write_checkpoint_column(group, series: pd.Series) -> None: import h5py + from microcosm.frame import nullable_boolean_values_and_mask + dtype = series.dtype group.attrs["pandas_dtype"] = str(dtype) if pd.api.types.is_bool_dtype(dtype): group.attrs["storage_kind"] = "bool" - values = series.to_numpy(dtype=np.bool_) + if isinstance(dtype, pd.BooleanDtype): + values, null_mask = nullable_boolean_values_and_mask(series) + group.attrs["nullable"] = True + group.attrs["has_null_mask"] = bool(null_mask.any()) + if null_mask.any(): + group.create_dataset( + "null_mask", + data=null_mask.astype(np.uint8, copy=False), + compression="gzip", + ) + else: + values = series.to_numpy(dtype=np.bool_, copy=False) + group.attrs["nullable"] = False + group.attrs["has_null_mask"] = False group.create_dataset("values", data=values, compression="gzip") elif pd.api.types.is_integer_dtype(dtype): if bool(series.isna().any()): @@ -2303,7 +2321,60 @@ def _read_checkpoint_column(group) -> np.ndarray: if storage_kind == "string": return np.asarray(dataset.asstr()[()], dtype=object) if storage_kind == "bool": - return np.asarray(dataset[()], dtype=np.bool_) + values = np.asarray(dataset[()]) + if values.ndim != 1 or values.dtype != np.dtype(np.bool_): + raise RuntimeError( + "Target-frame checkpoint boolean values must be a " + "one-dimensional bool array." + ) + nullable = group.attrs.get("nullable") + has_null_mask = group.attrs.get("has_null_mask") + if not isinstance(nullable, (bool, np.bool_)) or not isinstance( + has_null_mask, (bool, np.bool_) + ): + raise RuntimeError( + "Target-frame checkpoint boolean metadata must declare " + "nullable and has_null_mask booleans." + ) + nullable = bool(nullable) + has_null_mask = bool(has_null_mask) + if not nullable and has_null_mask: + raise RuntimeError( + "Native target-frame checkpoint boolean cannot carry a null mask." + ) + if has_null_mask: + if "null_mask" not in group: + raise RuntimeError( + "Nullable target-frame checkpoint boolean is missing its " + "declared null mask." + ) + null_mask = np.asarray(group["null_mask"]) + if ( + null_mask.ndim != 1 + or null_mask.dtype != np.dtype(np.uint8) + or len(null_mask) != len(values) + or ((null_mask != 0) & (null_mask != 1)).any() + or not null_mask.any() + ): + raise RuntimeError( + "Target-frame checkpoint nullable boolean null mask must " + "be an aligned, nonempty uint8 0/1 array." + ) + mask = null_mask.astype(np.bool_, copy=False) + if values[mask].any(): + raise RuntimeError( + "Target-frame checkpoint nullable boolean null mask covers " + "noncanonical true value bits." + ) + else: + if "null_mask" in group: + raise RuntimeError( + "Target-frame checkpoint boolean has an unexpected null mask." + ) + mask = np.zeros(len(values), dtype=np.bool_) + if nullable: + return pd.arrays.BooleanArray(values, mask, copy=False) + return values if storage_kind == "int64": return np.asarray(dataset[()], dtype=np.int64) if storage_kind == "float64": diff --git a/tools/build_us_multispine_pool.py b/tools/build_us_multispine_pool.py index 5a21f818..898a74d7 100644 --- a/tools/build_us_multispine_pool.py +++ b/tools/build_us_multispine_pool.py @@ -3,7 +3,7 @@ The default production path is the stacked pipeline: -``stack -> gap-fill -> PUF pass + tail -> derive -> seed -> simulate -> gates``. +``stack -> gap-fill -> PUF pass + tail -> late DAG -> derive -> seed -> simulate -> gates``. Both survey arms use one composition-preserving ``--sample-fraction``; PUF donors always remain full. The terminal completeness gate plus by-origin @@ -95,8 +95,10 @@ from microcosm.build.us_runtime.h5_io import ( US_MULTISPINE_AGREEMENT_DIAGNOSTICS_ARTIFACT_KIND, US_MULTISPINE_POOL_H5_ARTIFACT_KIND, + US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION, US_MULTISPINE_POOL_MANIFEST_ARTIFACT_KIND, US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION, + US_STACKED_POOL_OPERATOR_ORDER, load_simulation_ready_us_multispine_pool_manifest, write_nullable_us_h5, ) @@ -120,6 +122,7 @@ complete_multispine_source_inputs, derive_multispine_pool_inputs, materialize_multispine_agreement_outputs, + pool_remaining_stage_input_manifest_receipt, pool_transfer_target_families, prepare_multispine_source_inputs_for_clone, run_multispine_pool_path, @@ -129,6 +132,8 @@ assert_operator_free_source_frame, ) from microcosm.build.us_runtime.puf_capital_gains_tail import ( + PUF_CAPITAL_GAINS_TAIL_MANIFEST_SCHEMA_VERSION, + puf_capital_gains_tail_support_contract_identity, transfer_puf_capital_gains_tail, validate_puf_capital_gains_tail_manifest, ) @@ -158,13 +163,17 @@ by_origin_battery, gap_fill_stacked_spine, prepare_stacked_tail_derivation, + run_stacked_late_producer_dag, run_stacked_puf_pass, stacked_completeness_gate, stacked_gap_fill_plan, stacked_gap_fill_producer_schedule_receipt, + stacked_late_primary_checkpoint_input_binding, + stacked_late_primary_resource_receipts, + stacked_late_producer_resource_semantics_receipt, stacked_spine_authority_receipt, - transfer_stacked_post_puf_inputs, - validate_stacked_post_puf_transfer_receipt, + validate_stacked_late_producer_receipt, + validate_stacked_late_producer_transition_authority, validate_stacked_spine_frame, ) from microcosm.build.us_runtime.support_provenance import ( @@ -173,6 +182,10 @@ validate_assembly_provenance, ) from microcosm.build.us_runtime.take_up_contract import take_up_contract_identity +from microcosm.build.us_runtime.us_late_producer_registry import ( + CANONICAL_US_LATE_TRANSFER_GROUPS, + us_late_producer_schedule_receipt, +) from microcosm.frame import US_SCHEMA, Frame __all__ = [ @@ -191,6 +204,12 @@ POOL_MANIFEST_SCHEMA_VERSION = US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION """Schema version for the companion pool build manifest.""" +# ``--legacy-two-spine`` is a byte-stable compatibility surface. Stacked +# publication and checkpoint-envelope versions may advance without rewriting +# the retiring pipeline's last supported envelope. +_LEGACY_POOL_MANIFEST_SCHEMA_VERSION = 4 +_LEGACY_POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION = 3 + POOL_H5_ARTIFACT_KIND = US_MULTISPINE_POOL_H5_ARTIFACT_KIND """Neutral H5 artifact kind; readiness is asserted only by the manifest.""" @@ -206,6 +225,22 @@ # 2: The take-up contract identity binds the canonical SHA-256 of the entire # parsed resource, plus readable explicit fields. Version-1 volume # checkpoints are deliberately stale. +# 3: The tail-manifest schema and filing-status-exact recipient-support +# contract are explicit identity fields. Earlier checkpoints may have +# silently hard-failed a thin status and are deliberately stale. +# 4: The declared late-stage producer-input DAG, including its derived order, +# full source-input inventories, and nineteen bounded transfer groups, is +# bound into checkpoint identity. Fixed source-then-transfer checkpoints +# are deliberately stale. +# 5: Stacked transferred and simulated checkpoints carry and validate the +# independently propagated late-producer transition authority. Earlier +# envelopes cannot authenticate a reissued execution receipt. +# 6: Frame-checkpoint schema v3 materializes pandas nullable booleans as bool +# values plus an explicit null mask when needed. Earlier envelopes cannot +# prove that declared absences survived serialization. +# 7: Stacked primary-PUF output universes are explicit. Earlier envelopes can +# contain nulls outside the PUF clone for an output declared over the whole +# pool and therefore cannot resume safely even when their bank is reusable. # # Bump this version whenever any producer above changes a stage output without # changing one of the explicit identity fields below. In particular, adding, @@ -220,7 +255,7 @@ # normalizes that logical view in memory. Moving between those encodings does # not change a producer's scalar output and therefore does not advance this # ledger; changing string values or the canonical logical dtype policy does. -POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION = 2 +POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION = 7 _PRIMARY_QRF_N_ESTIMATORS = 100 _ACS_TRANSFER_N_ESTIMATORS = 100 @@ -241,16 +276,18 @@ _BANK_IDENTITY_SIBLING_SCAN_LIMIT = 64 _STACKED_SAMPLE_RUNG_TOKENS: Mapping[float, str] = { 0.01: "f001", + 0.04: "f004", 0.10: "f010", 1.00: "f100", } _STACKED_PIPELINE = "us-stacked-pool" -# Version 6 binds the ACS earnings-universe and whole-pool QBI reconciliation -# contracts, plus primary-QRF schema 6. Earlier checkpoints can carry stale -# recipient and mutation semantics and must rebuild. -_STACKED_CHECKPOINT_MATERIALIZER_VERSION = 6 +# Version 11 additionally binds the primary-PUF whole-pool universe semantics. +# Earlier checkpoints must rebuild rather than resume with a nullable +# s_corp_income leaf. Version 10 bound the complete late-resource semantics and +# corrected outer order (the primary PUF callback is nested inside the DAG). +_STACKED_CHECKPOINT_MATERIALIZER_VERSION = 11 _STACKED_RELEASE_ID_PATTERN = re.compile( - r"^populace-us-2024-stacked-f(?:001|010|100)-s[0-9]+-" + r"^populace-us-2024-stacked-f(?:001|004|010|100)-s[0-9]+-" r"asec[0-9]+-acs[0-9]+-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{8}$" ) @@ -281,6 +318,7 @@ class StackedPoolBuildResult: terminal_gates: tuple[GateResult, GateResult] release_id: str qbi_transition_authority_sha256: str | None = None + late_producer_transition_authority_sha256: str | None = None @property def simulation_ready(self) -> bool: @@ -885,13 +923,20 @@ def _pool_checkpoint_base_identity( verified_inputs: Mapping[str, _VerifiedInput], *, policyengine_us_version: str | None = None, + materializer_version: int | None = None, ) -> dict[str, object]: """Return every input and semantic surface that determines cached stages.""" + resolved_materializer_version = ( + POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION + if materializer_version is None + else materializer_version + ) + return { "artifact_kind": "populace_us_multispine_pool_checkpoint_identity", "schema_version": POOL_STAGE_CHECKPOINT_SCHEMA_VERSION, - "materializer_version": POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION, + "materializer_version": resolved_materializer_version, "period": POOL_TIME_PERIOD, "seed": POOL_RANDOM_SEED, "policyengine_us_version": ( @@ -915,10 +960,17 @@ def _pool_checkpoint_base_identity( "post_clone_source_operator_order": list( POOL_POST_CLONE_SOURCE_OPERATOR_ORDER ), + "late_producer_schedule": _json_ready(us_late_producer_schedule_receipt()), "derive_operator_order": list(POOL_DERIVE_OPERATOR_ORDER), "primary_qrf_target_order": list(PRIMARY_QRF_TARGET_ORDER), "transfer_target_families": _json_ready(pool_transfer_target_families()), "take_up_contract": take_up_contract_identity(), + "puf_capital_gains_tail_manifest_schema_version": ( + PUF_CAPITAL_GAINS_TAIL_MANIFEST_SCHEMA_VERSION + ), + "puf_capital_gains_tail_support_contract": ( + puf_capital_gains_tail_support_contract_identity() + ), "primary_qrf_n_estimators": _PRIMARY_QRF_N_ESTIMATORS, "acs_transfer_n_estimators": _ACS_TRANSFER_N_ESTIMATORS, "acs_transfer_max_targets_per_fit": ( @@ -929,12 +981,29 @@ def _pool_checkpoint_base_identity( } +def _legacy_pool_checkpoint_base_identity( + verified_inputs: Mapping[str, _VerifiedInput], + *, + policyengine_us_version: str | None = None, +) -> dict[str, object]: + """Return the retiring pipeline identity without stacked-only DAG state.""" + + identity = _pool_checkpoint_base_identity( + verified_inputs, + policyengine_us_version=policyengine_us_version, + materializer_version=_LEGACY_POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION, + ) + pool_code = dict(identity["pool_code"]) + del pool_code["late_producer_schedule"] + return {**identity, "pool_code": pool_code} + + def _stacked_rung(sample_fraction: float) -> str: try: return _STACKED_SAMPLE_RUNG_TOKENS[float(sample_fraction)] except KeyError as exc: raise ValueError( - "Stacked sample_fraction must be one standard rung: 0.01, 0.10, or 1.0." + "Stacked sample_fraction must be one standard rung: 0.01, 0.04, 0.10, or 1.0." ) from exc @@ -1022,20 +1091,7 @@ def _stacked_checkpoint_base_identity( }, "stacked_authority": stacked_spine_authority_receipt(), "pool_code": { - "operator_order": [ - "assemble_stacked_spine", - "prepare_multispine_source_inputs_for_clone", - "gap_fill_stacked_spine", - "run_stacked_puf_pass", - "complete_multispine_source_inputs", - "transfer_stacked_post_puf_inputs", - "prepare_stacked_tail_derivation", - "derive_multispine_pool_inputs", - "seed_multispine_pool_inputs", - "materialize_multispine_agreement_outputs", - "stacked_completeness_gate", - "by_origin_battery", - ], + "operator_order": list(US_STACKED_POOL_OPERATOR_ORDER), "pre_clone_source_operator_order": list( POOL_PRE_CLONE_SOURCE_OPERATOR_ORDER ), @@ -1045,7 +1101,24 @@ def _stacked_checkpoint_base_identity( "post_clone_source_operator_order": list( POOL_POST_CLONE_SOURCE_OPERATOR_ORDER ), + "late_producer_schedule": _json_ready(us_late_producer_schedule_receipt()), + "late_producer_resource_semantics": ( + stacked_late_producer_resource_semantics_receipt( + clone_attachment_fraction=clone_attachment_fraction, + clone_attachment_seed=clone_attachment_seed, + primary_seed=POOL_RANDOM_SEED, + primary_n_estimators=_PRIMARY_QRF_N_ESTIMATORS, + transfer_seed=POOL_RANDOM_SEED, + transfer_n_estimators=_ACS_TRANSFER_N_ESTIMATORS, + transfer_max_targets_per_fit=( + DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT + ), + ) + ), "derive_operator_order": list(POOL_DERIVE_OPERATOR_ORDER), + "remaining_stage_input_manifest": ( + pool_remaining_stage_input_manifest_receipt() + ), "primary_qrf_target_order": list(PRIMARY_QRF_TARGET_ORDER), "primary_qrf_checkpoint_schema_version": ( PRIMARY_QRF_CHECKPOINT_SCHEMA_VERSION @@ -1057,6 +1130,12 @@ def _stacked_checkpoint_base_identity( us_qbi_reconciliation_contract_identity() ), "take_up_contract": take_up_contract_identity(), + "puf_capital_gains_tail_manifest_schema_version": ( + PUF_CAPITAL_GAINS_TAIL_MANIFEST_SCHEMA_VERSION + ), + "puf_capital_gains_tail_support_contract": ( + puf_capital_gains_tail_support_contract_identity() + ), "primary_qrf_n_estimators": _PRIMARY_QRF_N_ESTIMATORS, "acs_transfer_n_estimators": _ACS_TRANSFER_N_ESTIMATORS, "acs_transfer_max_targets_per_fit": ( @@ -1250,6 +1329,7 @@ def __init__( root: Path, *, base_identity: Mapping[str, object], + materializer_version: int | None = None, ) -> None: self.root = Path(root) if self.root.exists() and not self.root.is_dir(): @@ -1259,7 +1339,21 @@ def __init__( normalized_identity = _json_ready(base_identity) if not isinstance(normalized_identity, dict): # pragma: no cover raise TypeError("Pool checkpoint base identity must be an object.") + resolved_materializer_version = ( + POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION + if materializer_version is None + else materializer_version + ) + if ( + isinstance(resolved_materializer_version, bool) + or not isinstance(resolved_materializer_version, int) + or resolved_materializer_version < 1 + ): + raise ValueError( + "Pool checkpoint materializer_version must be a positive integer." + ) self._base_identity = normalized_identity + self._materializer_version = resolved_materializer_version self._input_receipts: dict[str, object] | None = None self._resumed_from: str | None = None self._attempts: dict[str, dict[str, object]] = { @@ -1347,6 +1441,16 @@ def write(self, checkpoint: MultispinePoolCheckpoint) -> None: boundary=f"pool {stage} checkpoint write", ) qbi_route = _checkpoint_qbi_route(self._base_identity) + if stage in {"transferred", "simulated"} and qbi_route == "stacked": + _validate_stacked_post_puf_stage_receipt( + persistent_frame, + checkpoint.stage_receipts, + boundary=f"pool {stage} durable checkpoint write", + transition_authority_sha256=( + checkpoint.late_producer_transition_authority_sha256 + ), + require_live_output=stage == "transferred", + ) if stage == "simulated" and qbi_route is not None: _validate_qbi_stage_receipt( persistent_frame, @@ -1396,7 +1500,7 @@ def write(self, checkpoint: MultispinePoolCheckpoint) -> None: metadata = { "artifact_kind": _POOL_STAGE_CHECKPOINT_ARTIFACT_KIND, "schema_version": POOL_STAGE_CHECKPOINT_SCHEMA_VERSION, - "materializer_version": POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION, + "materializer_version": self._materializer_version, "stage": stage, "identity": identity, "identity_sha256": identity_sha256, @@ -1420,6 +1524,10 @@ def write(self, checkpoint: MultispinePoolCheckpoint) -> None: metadata["qbi_transition_authority_sha256"] = ( checkpoint.qbi_transition_authority_sha256 ) + if checkpoint.late_producer_transition_authority_sha256 is not None: + metadata["late_producer_transition_authority_sha256"] = ( + checkpoint.late_producer_transition_authority_sha256 + ) path = self.checkpoint_path(stage) started_at = time.perf_counter() write_frame_checkpoint(path, stored_frame, metadata=metadata) @@ -1431,7 +1539,7 @@ def write(self, checkpoint: MultispinePoolCheckpoint) -> None: { "artifact_kind": _POOL_STAGE_CHECKPOINT_MANIFEST_ARTIFACT_KIND, "schema_version": POOL_STAGE_CHECKPOINT_SCHEMA_VERSION, - "materializer_version": POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION, + "materializer_version": self._materializer_version, "stage": stage, "identity": identity, "identity_sha256": identity_sha256, @@ -1452,6 +1560,15 @@ def write(self, checkpoint: MultispinePoolCheckpoint) -> None: if "qbi_transition_authority_sha256" in metadata else {} ), + **( + { + "late_producer_transition_authority_sha256": metadata[ + "late_producer_transition_authority_sha256" + ] + } + if "late_producer_transition_authority_sha256" in metadata + else {} + ), }, ) receipts_record: dict[str, object] = { @@ -1464,9 +1581,7 @@ def write(self, checkpoint: MultispinePoolCheckpoint) -> None: { "artifact_kind": _POOL_STAGE_CHECKPOINT_RECEIPTS_ARTIFACT_KIND, "schema_version": POOL_STAGE_CHECKPOINT_SCHEMA_VERSION, - "materializer_version": ( - POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION - ), + "materializer_version": self._materializer_version, "stage": stage, "identity_sha256": identity_sha256, "checkpoint": { @@ -1533,7 +1648,7 @@ def provenance( ), "identity_sha256": _pool_checkpoint_identity_sha256(identity), "schema_version": POOL_STAGE_CHECKPOINT_SCHEMA_VERSION, - "materializer_version": (POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION), + "materializer_version": self._materializer_version, "path": str(self.checkpoint_path(artifact_stage).resolve()), "manifest_path": str( self.checkpoint_manifest_path(artifact_stage).resolve() @@ -1570,7 +1685,7 @@ def provenance( return { "artifact_kind": "populace_us_multispine_pool_checkpoint_provenance", "schema_version": POOL_STAGE_CHECKPOINT_SCHEMA_VERSION, - "materializer_version": POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION, + "materializer_version": self._materializer_version, "root": str(self.root.resolve()), "base_identity_sha256": self.base_identity_sha256, "deepest_resumed_stage": self._resumed_from, @@ -1624,8 +1739,7 @@ def _load(self, stage: str) -> MultispinePoolCheckpoint | None: != _POOL_STAGE_CHECKPOINT_MANIFEST_ARTIFACT_KIND or manifest.get("schema_version") != POOL_STAGE_CHECKPOINT_SCHEMA_VERSION - or manifest.get("materializer_version") - != POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION + or manifest.get("materializer_version") != self._materializer_version or manifest.get("stage") != stage ): raise ValueError( @@ -1697,12 +1811,14 @@ def _load(self, stage: str) -> MultispinePoolCheckpoint | None: stage=stage, expected_identity=expected_identity, expected_identity_sha256=expected_identity_sha256, + expected_materializer_version=self._materializer_version, ) for key in ( "row_counts", "frame_schema", "frame_metadata", "qbi_transition_authority_sha256", + "late_producer_transition_authority_sha256", ): if metadata.get(key) != manifest.get(key): raise ValueError( @@ -1730,6 +1846,9 @@ def _load(self, stage: str) -> MultispinePoolCheckpoint | None: qbi_transition_authority_sha256 = metadata.get( "qbi_transition_authority_sha256" ) + late_producer_transition_authority_sha256 = metadata.get( + "late_producer_transition_authority_sha256" + ) input_receipts = metadata.get("input_receipts") if not isinstance(assembly_receipt, Mapping): raise ValueError( @@ -1771,6 +1890,16 @@ def _load(self, stage: str) -> MultispinePoolCheckpoint | None: persistent_frame = _without_simulation_output(frame) qbi_route = _checkpoint_qbi_route(self._base_identity) + if stage in {"transferred", "simulated"} and qbi_route == "stacked": + _validate_stacked_post_puf_stage_receipt( + persistent_frame, + restored_stage_receipts, + boundary=f"pool {stage} durable checkpoint load", + transition_authority_sha256=( + late_producer_transition_authority_sha256 + ), + require_live_output=stage == "transferred", + ) if stage == "simulated" and qbi_route is not None: _validate_qbi_stage_receipt( persistent_frame, @@ -1787,6 +1916,9 @@ def _load(self, stage: str) -> MultispinePoolCheckpoint | None: stage_receipts=restored_stage_receipts, simulation_frame=simulation_frame, qbi_transition_authority_sha256=(qbi_transition_authority_sha256), + late_producer_transition_authority_sha256=( + late_producer_transition_authority_sha256 + ), ) self.bind_input_receipts(input_receipts) self._attempts[stage] = { @@ -1845,8 +1977,7 @@ def _load_operational_stage_receipts( payload.get("artifact_kind") != _POOL_STAGE_CHECKPOINT_RECEIPTS_ARTIFACT_KIND or payload.get("schema_version") != POOL_STAGE_CHECKPOINT_SCHEMA_VERSION - or payload.get("materializer_version") - != POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION + or payload.get("materializer_version") != self._materializer_version or payload.get("stage") != stage or payload.get("identity_sha256") != expected_identity_sha256 ): @@ -1935,12 +2066,12 @@ def _validate_checkpoint_metadata( stage: str, expected_identity: Mapping[str, object], expected_identity_sha256: str, + expected_materializer_version: int, ) -> None: if ( metadata.get("artifact_kind") != _POOL_STAGE_CHECKPOINT_ARTIFACT_KIND or metadata.get("schema_version") != POOL_STAGE_CHECKPOINT_SCHEMA_VERSION - or metadata.get("materializer_version") - != POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION + or metadata.get("materializer_version") != expected_materializer_version or metadata.get("stage") != stage ): raise ValueError(f"{stage} checkpoint metadata has an unsupported binding") @@ -2478,13 +2609,26 @@ def _stacked_direction_bank_identity( } -def _stacked_post_puf_bank_identity( +def _stacked_late_producer_bank_identity( checkpoint_identity: Mapping[str, object], + *, + producer_name: str, + entity: str, + family: str, + ordered_targets: tuple[str, ...], ) -> dict[str, object]: + schedule = us_late_producer_schedule_receipt() return { **_pool_checkpoint_stage_identity(checkpoint_identity, "transferred"), - "stacked_transfer_stage": "post_puf_source_completion", - "stacked_transfer_name": "asec_clone_1_to_missing", + "stacked_transfer_stage": "late_producer_dag", + "late_producer_dag_sha256": schedule["schedule_sha256"], + "late_producer_schedule_sha256": schedule["payload_sha256"], + "late_producer": { + "name": producer_name, + "entity": entity, + "family": family, + "ordered_targets": list(ordered_targets), + }, } @@ -2504,27 +2648,74 @@ def _stacked_tail_manifest( def _validate_stacked_post_puf_stage_receipt( + frame: Frame, stage_receipts: Mapping[str, Mapping[str, object]], *, boundary: str, + transition_authority_sha256: str | None, + require_live_output: bool, ) -> None: - """Require the nested late-transfer receipt to carry canonical authority.""" + """Require the complete DAG proof and both exact compatibility aliases.""" impute = stage_receipts.get("impute") if not isinstance(impute, Mapping): raise ValueError( f"{boundary}: stacked transferred receipts have no impute object." ) + dag_receipt = impute.get("stacked_late_producer_dag") + if not isinstance(dag_receipt, Mapping): + raise ValueError( + f"{boundary}: stacked transferred receipts have no late-producer " + "DAG object." + ) + # Authenticate the canonical execution proof before consulting the + # independently propagated live-frame anchor. This keeps malformed DAGs + # attributable to their receipt defect even when their authority carrier + # is also absent or corrupt. + validate_stacked_late_producer_receipt(dag_receipt, boundary=boundary) + if not isinstance(transition_authority_sha256, str): + raise ValueError( + f"{boundary}: independently carried late-producer transition " + "authority is absent." + ) + if require_live_output: + validate_stacked_late_producer_receipt( + dag_receipt, + boundary=boundary, + frame=frame, + expected_transition_authority_sha256=(transition_authority_sha256), + ) + else: + validate_stacked_late_producer_transition_authority( + frame, + dag_receipt, + boundary=boundary, + expected_transition_authority_sha256=(transition_authority_sha256), + ) transfer_receipt = impute.get("stacked_post_puf_transfer") if not isinstance(transfer_receipt, Mapping): raise ValueError( f"{boundary}: stacked transferred receipts have no post-PUF " "transfer object." ) - validate_stacked_post_puf_transfer_receipt( - transfer_receipt, - boundary=boundary, + if _json_ready(transfer_receipt) != _json_ready( + dag_receipt.get("post_puf_transfer") + ): + raise ValueError( + f"{boundary}: stacked post-PUF transfer alias differs from the " + "late-producer DAG proof." + ) + source_chain = impute.get("source_operator_chain") + source_alias = ( + source_chain.get("late_dag_completion") + if isinstance(source_chain, Mapping) + else None ) + if _json_ready(source_alias) != _json_ready(dag_receipt.get("source_completion")): + raise ValueError( + f"{boundary}: stacked source-completion alias differs from the " + "late-producer DAG proof." + ) def _qbi_receipt_from_stage_receipts( @@ -2606,11 +2797,15 @@ def _emit_stacked_checkpoint( stage_receipts: Mapping[str, Mapping[str, object]], simulation_frame: Frame | None = None, qbi_transition_authority_sha256: str | None = None, + late_producer_transition_authority_sha256: str | None = None, ) -> None: if stage in {"transferred", "simulated"}: _validate_stacked_post_puf_stage_receipt( + frame, stage_receipts, boundary=f"stacked {stage} checkpoint emission", + transition_authority_sha256=(late_producer_transition_authority_sha256), + require_live_output=stage == "transferred", ) if stage == "simulated": _validate_qbi_stage_receipt( @@ -2632,6 +2827,9 @@ def _emit_stacked_checkpoint( }, simulation_frame=simulation_frame, qbi_transition_authority_sha256=(qbi_transition_authority_sha256), + late_producer_transition_authority_sha256=( + late_producer_transition_authority_sha256 + ), ) ) @@ -2677,6 +2875,7 @@ def mark_phase(name: str) -> None: assembly_receipt = current.metadata[SPINE_ASSEMBLY_MANIFEST_KEY] receipts: dict[str, Mapping[str, object]] = {} qbi_transition_authority_sha256: str | None = None + late_producer_transition_authority_sha256: str | None = None resume_stage: str | None = None _emit_stacked_checkpoint( checkpoint, @@ -2709,11 +2908,17 @@ def mark_phase(name: str) -> None: name: dict(receipt) for name, receipt in resume.stage_receipts.items() } qbi_transition_authority_sha256 = resume.qbi_transition_authority_sha256 + late_producer_transition_authority_sha256 = ( + resume.late_producer_transition_authority_sha256 + ) resume_stage = resume.stage if resume_stage in {"transferred", "simulated"}: _validate_stacked_post_puf_stage_receipt( + current, receipts, boundary=f"stacked {resume_stage} checkpoint resume", + transition_authority_sha256=(late_producer_transition_authority_sha256), + require_live_output=resume_stage == "transferred", ) if resume_stage == "simulated": _validate_qbi_stage_receipt( @@ -2792,19 +2997,79 @@ def mark_phase(name: str) -> None: primary_qrf_checkpoint_dir, current_base_identity_sha256=current_base_identity_sha256, ) - puf_result = run_stacked_puf_pass( - gap_filled.frame, + late_target_banks = { + group.name: AcsTransferTargetBankStore( + acs_transfer_checkpoint_dir + / "late_producer_dag" + / group.entity + / group.family, + identity=_stacked_late_producer_bank_identity( + checkpoint_identity, + producer_name=group.name, + entity=group.entity, + family=group.family, + ordered_targets=group.targets, + ), + ) + for group in CANONICAL_US_LATE_TRANSFER_GROUPS + } + + def primary_puf_producer(primary_input: Frame): + produced = run_stacked_puf_pass( + primary_input, + puf_donor, + clone_attachment_fraction=clone_attachment_fraction, + clone_attachment_seed=clone_attachment_seed, + seed=POOL_RANDOM_SEED, + n_estimators=_PRIMARY_QRF_N_ESTIMATORS, + fit_records=fit_records, + tail_bound_diagnostics=tail_bound_diagnostics, + primary_qrf_checkpoint_dir=primary_qrf_checkpoint_dir, + primary_qrf_input_binding=primary_qrf_input_binding, + ) + produced_tail = produced.receipt.get("puf_capital_gains_tail_transfer") + if not isinstance(produced_tail, Mapping): + raise ValueError("Stacked PUF pass emitted no tail manifest.") + validate_puf_capital_gains_tail_manifest(produced_tail) + if not isinstance(produced.receipt.get("primary_puf_qrf"), Mapping): + raise ValueError("Stacked PUF pass emitted no primary-QRF receipt.") + mark_phase("puf_passed") + return produced + + primary_resource_receipts = stacked_late_primary_resource_receipts( puf_donor, + primary_qrf_checkpoint_identity_sha256=(current_base_identity_sha256), clone_attachment_fraction=clone_attachment_fraction, clone_attachment_seed=clone_attachment_seed, seed=POOL_RANDOM_SEED, n_estimators=_PRIMARY_QRF_N_ESTIMATORS, - fit_records=fit_records, - tail_bound_diagnostics=tail_bound_diagnostics, - primary_qrf_checkpoint_dir=primary_qrf_checkpoint_dir, + fit_records_enabled=True, + tail_bound_diagnostics_enabled=True, ) - mark_phase("puf_passed") - + primary_qrf_input_binding = stacked_late_primary_checkpoint_input_binding( + primary_resource_receipts + ) + late_stage = run_stacked_late_producer_dag( + gap_filled.frame, + primary_puf_producer=primary_puf_producer, + primary_resource_receipts=primary_resource_receipts, + seed=POOL_RANDOM_SEED, + n_estimators=_ACS_TRANSFER_N_ESTIMATORS, + max_targets_per_fit=DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, + target_banks=late_target_banks, + ) + late_producer_transition_authority_sha256 = ( + late_stage.transition_authority_sha256 + ) + validate_stacked_late_producer_receipt( + late_stage.receipt, + boundary="stacked cold-build late-producer DAG", + frame=late_stage.frame, + expected_transition_authority_sha256=( + late_producer_transition_authority_sha256 + ), + ) + puf_result = late_stage.primary_puf_result puf_receipt = dict(puf_result.receipt) primary_qrf_receipt = puf_receipt.pop("primary_puf_qrf") if not isinstance(primary_qrf_receipt, Mapping): @@ -2836,40 +3101,24 @@ def mark_phase(name: str) -> None: if not isinstance(tail_manifest, Mapping): raise ValueError("Stacked PUF pass emitted no tail manifest.") validate_puf_capital_gains_tail_manifest(tail_manifest) - - source_completion = complete_multispine_source_inputs(puf_result.frame) - completion_preservation = assert_stacked_tail_cells_preserved( - source_completion.frame, - tail_manifest, - ) - post_puf_target_bank = AcsTransferTargetBankStore( - acs_transfer_checkpoint_dir / "post_puf_transfer", - identity=_stacked_post_puf_bank_identity(checkpoint_identity), - ) - post_puf_transfer = transfer_stacked_post_puf_inputs( - source_completion.frame, - seed=POOL_RANDOM_SEED, - n_estimators=_ACS_TRANSFER_N_ESTIMATORS, - max_targets_per_fit=DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, - target_bank=post_puf_target_bank, - ) - validate_stacked_post_puf_transfer_receipt( - post_puf_transfer.receipt, - boundary="stacked cold-build post-PUF transfer", - ) - fit_records.extend(post_puf_transfer.transfer_result.fit_records) + post_puf_transfer_receipt = late_stage.receipt.get("post_puf_transfer") + if not isinstance(post_puf_transfer_receipt, Mapping): + raise ValueError( + "Stacked late-producer DAG emitted no post-PUF transfer receipt." + ) + fit_records.extend(late_stage.transfer_result.fit_records) weights_audit = weights_audit_gate(fit_records) if not weights_audit.passed: raise ValueError( "Stacked imputation weights audit failed:\n " + "\n ".join(weights_audit.failures) ) - post_puf_preservation = assert_stacked_tail_cells_preserved( - post_puf_transfer.frame, + late_stage_preservation = assert_stacked_tail_cells_preserved( + late_stage.frame, tail_manifest, ) current = canonicalize_frame_string_dtypes( - post_puf_transfer.frame, + late_stage.frame, boundary="stacked pool transferred checkpoint", in_place=True, ) @@ -2880,15 +3129,15 @@ def mark_phase(name: str) -> None: receipts["impute"] = { "source_operator_chain": { "pre_gap_fill_preparation": dict(prepared.receipt), - "post_primary_completion": dict(source_completion.receipt), + "late_dag_completion": dict(late_stage.source_completion_receipt), }, "stacked_gap_fill": dict(gap_filled.receipt), - "stacked_post_puf_transfer": dict(post_puf_transfer.receipt), + "stacked_late_producer_dag": dict(late_stage.receipt), + "stacked_post_puf_transfer": dict(post_puf_transfer_receipt), "primary_puf_qrf": primary_qrf_receipt, "puf_capital_gains_tail_transfer": dict(tail_manifest), "stacked_puf_pass": puf_receipt, - "tail_preservation_after_source_completion": completion_preservation, - "tail_preservation_after_post_puf_transfer": post_puf_preservation, + "tail_preservation_after_late_producer_dag": late_stage_preservation, "acs_qrf_transfer": { "target_families": { "early_gap_fill": _json_ready(CANONICAL_STACKED_GAP_FILL_SURFACE), @@ -2904,7 +3153,10 @@ def mark_phase(name: str) -> None: name: bank.receipt() for name, bank in sorted(target_banks.items()) }, - "post_puf_transfer": post_puf_target_bank.receipt(), + "late_producer_groups": { + name: bank.receipt() + for name, bank in sorted(late_target_banks.items()) + }, }, }, "weights_audit": GateReport((weights_audit,)).to_manifest(), @@ -2915,6 +3167,9 @@ def mark_phase(name: str) -> None: frame=current, assembly_receipt=assembly_receipt, stage_receipts=receipts, + late_producer_transition_authority_sha256=( + late_producer_transition_authority_sha256 + ), ) mark_phase("transferred") else: @@ -2987,6 +3242,9 @@ def mark_phase(name: str) -> None: stage_receipts=receipts, simulation_frame=simulation_frame, qbi_transition_authority_sha256=(qbi_transition_authority_sha256), + late_producer_transition_authority_sha256=( + late_producer_transition_authority_sha256 + ), ) mark_phase("simulated") else: @@ -3002,8 +3260,14 @@ def mark_phase(name: str) -> None: ) assert_stacked_tail_cells_preserved(simulation_frame, tail_manifest) - completeness = stacked_completeness_gate(simulation_frame) - battery = by_origin_battery(simulation_frame) + completeness = stacked_completeness_gate( + simulation_frame, + tail_manifest=tail_manifest, + ) + battery = by_origin_battery( + simulation_frame, + tail_manifest=tail_manifest, + ) # Manifest conversion is itself the final canonical-authority check and # deliberately happens before publication or readiness is asserted. GateReport((completeness, battery)).to_manifest() @@ -3021,6 +3285,9 @@ def mark_phase(name: str) -> None: terminal_gates=(completeness, battery), release_id=release_id, qbi_transition_authority_sha256=qbi_transition_authority_sha256, + late_producer_transition_authority_sha256=( + late_producer_transition_authority_sha256 + ), ) @@ -3051,7 +3318,7 @@ def _manifest_payload( raise ValueError("Pool input receipts have no PUF donor object.") return { "artifact_kind": "populace_us_multispine_pool_manifest", - "schema_version": POOL_MANIFEST_SCHEMA_VERSION, + "schema_version": _LEGACY_POOL_MANIFEST_SCHEMA_VERSION, "status": status, "simulation_ready": result.simulation_ready, "publication_run_id": publication_run_id, @@ -3133,8 +3400,11 @@ def _stacked_manifest_payload( """Build the stacked-only manifest without changing the legacy envelope.""" _validate_stacked_post_puf_stage_receipt( + result.frame, result.stage_receipts, boundary="stacked production manifest", + transition_authority_sha256=(result.late_producer_transition_authority_sha256), + require_live_output=False, ) _validate_qbi_stage_receipt( result.frame, @@ -3158,20 +3428,7 @@ def _stacked_manifest_payload( "simulation_ready": result.simulation_ready, "publication_run_id": publication_run_id, "calibration_applied": False, - "operator_order": [ - "assemble_stacked_spine", - "prepare_multispine_source_inputs_for_clone", - "gap_fill_stacked_spine", - "run_stacked_puf_pass", - "complete_multispine_source_inputs", - "transfer_stacked_post_puf_inputs", - "prepare_stacked_tail_derivation", - "derive_multispine_pool_inputs", - "seed_multispine_pool_inputs", - "materialize_multispine_agreement_outputs", - "stacked_completeness_gate", - "by_origin_battery", - ], + "operator_order": list(US_STACKED_POOL_OPERATOR_ORDER), "period": POOL_TIME_PERIOD, "random_seed": POOL_RANDOM_SEED, "sampling": { @@ -3196,6 +3453,9 @@ def _stacked_manifest_payload( role: pin.to_manifest() for role, pin in verified_inputs.items() }, "input_pins_digest": _input_pins_digest(verified_inputs), + "late_producer_transition_authority_sha256": ( + result.late_producer_transition_authority_sha256 + ), "asec_raw_stage_checkpoint": input_receipts.get("asec_raw_stage_checkpoint"), "acs_source_manifest": asdict(acs_source_manifest), "acs_pums_build": input_receipts.get("acs_pums_build"), @@ -3227,6 +3487,7 @@ def _stacked_manifest_payload( "size_bytes": outputs.pool_h5.stat().st_size, "artifact_kind": POOL_H5_ARTIFACT_KIND, "publication_run_id": publication_run_id, + "materializer_version": US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION, "nullable": True, "input_only": True, "formula_outputs_persisted": False, @@ -3270,6 +3531,7 @@ def _stacked_publication_tombstone( "path": str(outputs.pool_h5.resolve()), "artifact_kind": POOL_H5_ARTIFACT_KIND, "publication_run_id": publication_run_id, + "materializer_version": US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION, }, "agreement_diagnostics": { "path": str(outputs.agreement_diagnostics.resolve()), @@ -3286,7 +3548,7 @@ def _publication_tombstone( ) -> dict[str, object]: return { "artifact_kind": US_MULTISPINE_POOL_MANIFEST_ARTIFACT_KIND, - "schema_version": POOL_MANIFEST_SCHEMA_VERSION, + "schema_version": _LEGACY_POOL_MANIFEST_SCHEMA_VERSION, "status": "publication_in_progress", "simulation_ready": False, "publication_run_id": publication_run_id, @@ -3339,7 +3601,9 @@ def _write_outputs( checkpoint_provenance = { "artifact_kind": ("populace_us_multispine_pool_checkpoint_provenance"), "schema_version": POOL_STAGE_CHECKPOINT_SCHEMA_VERSION, - "materializer_version": POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION, + "materializer_version": ( + _LEGACY_POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION + ), "enabled": False, "agreement": { "source": "always_fresh", @@ -3365,7 +3629,7 @@ def _write_outputs( ) diagnostics = { "artifact_kind": US_MULTISPINE_AGREEMENT_DIAGNOSTICS_ARTIFACT_KIND, - "schema_version": POOL_MANIFEST_SCHEMA_VERSION, + "schema_version": _LEGACY_POOL_MANIFEST_SCHEMA_VERSION, "simulation_ready": result.simulation_ready, "publication_run_id": publication_run_id, "agreement_gate": _agreement_payload(result), @@ -3412,8 +3676,11 @@ def _write_stacked_outputs( """Atomically publish the stacked input-only pool and terminal receipts.""" _validate_stacked_post_puf_stage_receipt( + result.frame, result.stage_receipts, boundary="stacked publication entry", + transition_authority_sha256=(result.late_producer_transition_authority_sha256), + require_live_output=False, ) _validate_qbi_stage_receipt( result.frame, @@ -3458,6 +3725,7 @@ def _write_stacked_outputs( period=POOL_TIME_PERIOD, artifact_kind=POOL_H5_ARTIFACT_KIND, publication_run_id=publication_run_id, + materializer_version=US_MULTISPINE_POOL_H5_MATERIALIZER_VERSION, ) _atomic_write_json(temporary_diagnostics, diagnostics) os.replace(temporary_h5, outputs.pool_h5) @@ -3579,7 +3847,8 @@ def _main_legacy(args: argparse.Namespace) -> int: verified_inputs, acs_source_manifest = _verify_inputs(args, outputs) checkpoint_store = _PoolStageCheckpointStore( outputs.checkpoint_root, - base_identity=_pool_checkpoint_base_identity(verified_inputs), + base_identity=_legacy_pool_checkpoint_base_identity(verified_inputs), + materializer_version=_LEGACY_POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION, ) outputs = _with_checkpoint_identity( outputs, @@ -3959,6 +4228,7 @@ def _main_stacked(args: argparse.Namespace) -> int: checkpoint_store = _PoolStageCheckpointStore( outputs.checkpoint_root, base_identity=checkpoint_identity, + materializer_version=POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION, ) outputs = _with_checkpoint_identity( outputs, @@ -4033,6 +4303,7 @@ def _main_stacked(args: argparse.Namespace) -> int: checkpoint_store = _PoolStageCheckpointStore( outputs.checkpoint_root, base_identity=checkpoint_identity, + materializer_version=POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION, ) outputs = _with_checkpoint_identity( outputs, diff --git a/tools/logbook.py b/tools/logbook.py index 0d41edd1..d02e778e 100644 --- a/tools/logbook.py +++ b/tools/logbook.py @@ -94,7 +94,7 @@ def _parser() -> argparse.ArgumentParser: ) render.add_argument( "--rung", - help="Include only this Logbook rung token (f001, f010, or f100).", + help="Include only this Logbook rung token (f001, f004, f010, or f100).", ) render.add_argument( "--disposition",