M6 harness: source person sex canonically in build_realized_population (reg-3 crash-2 fix + schema audit)

[MaxGhenis]

What and why

The third registered gate_m6 candidate-1 run (registration 4971244215) failed
to execute with two pre-scoring crashes; the second was an integration defect,
graded in #42 comment 4972045579:

ValueError: anchor source is missing columns ['sex']   (m6_population.py:139)

build_realized_population read sex from the demographic panel via
_anchor_rows(columns=("age","sex","interview")) (m6_population.py:186), but
the certified panels.demographic_panel carries seven columns and no sex:
{person_id, period, age, sequence, relationship, weight, interview}
(panels.py:252-253). The only test over the path used a synthetic fixture that
baked sex into the demo frame, so the unit suite passed while the first real
execution of the phase (m6_runner.py:355) could not. Run 1 masked it (its
QRF-import crash at m6_runner.py:345 precedes :355).

The canonical sex source (with certified-code citations)

Person sex is ER32000 from the PSID cross-year individual file, read
person-constant by data.deaths.read_death_records (deaths.py:171-244). This
is the single canonical attach the certified builders use whenever
person-constant sex is joined to the demographic-panel population:

• data.household_composition.join_demographics merges sex from the
  death-records frame by person_id (household_composition.py:400), and
  build_household_panel sources it from deaths.read_death_records
  (household_composition.py:446-449);
• data.disability.attach_sex maps sex person-by-person from the same death
  records (disability.py:443-446).

§2.8.3 already names data.deaths as the sex source "for the mortality slices"
(line 1007). Marriage-file sex (marriage.marriage_history, MH4) is
marriage-file-scoped — a subset universe used only inside the marital subsystem —
so it is not a competing source for the anchor persons. There was exactly one
canonical pattern to mirror; no design choice was made here.

Fix: build_realized_population now takes the death_records frame and joins
person sex by person_id before the demographic seed (_person_sex_map +
.assign), validating uniqueness (one coded value per person; conflicting
rows raise) and full coded-sex coverage over the anchor persons (any anchor
person without a male/female sex raises). The runner passes
inputs.death_records, which M6HarnessInputs already carries.

Real-frame proof (population construction → build_realized_population)

Ran on the staged PSID frames (population construction precedes the QRF refit, so
no fitting stack / pe-us needed; no pre-flights, scoring, or artifact writes):

quantity	value
demographic_panel rows	892,639 (7 cols, no sex)
death_records rows	85,536
holdout persons (anchor)	29,792
initial slice (2015 bulk)	23,094
scheduled openers	2017: 4,571 · 2019: 2,127
anchor persons carrying sex	29,792 (100%)
seed sex distribution	female 15,499 · male 14,293
seed sex nulls	0

build_realized_population completes; every anchor person has a coded sex, so the
full-coverage invariant holds on real data — crash-2 is resolved.

Schema audit (the check whose absence let run 2 crash)

New harness/m6_schema_audit.py: an explicit in-code manifest of every column
each real-frame-reading phase reads from each certified frame, plus the committed
loader schemas and an audit_columns function. Scope: 8 phases
(assemble_m6_inputs require-columns, build_anchor_frame, presence_by_wave,
mortality_slices, marital_tables_from_panel, disability_pairs,
earnings_frame, build_realized_population) × 9 frames
(demographic_panel, death_records, earnings_panel, disability_status,
marital.person_years, marital.events, household.person_waves,
cohabitation, disability_panel.person_years).

• unit tier (tests/test_m6_schema_audit.py): manifest vs committed schemas,
  plus a discriminating test that a phantom read is flagged (green ≠ vacuous);
• integration_psid tier (tests/test_m6_schema_audit_psid.py): manifest vs
  the real loaders' columns, plus an exact committed-vs-real drift check —
  both green against staged PSID (committed schemas reconciled to reality:
  disability_status also emits sequence/status; marital.events emits
  origin).

The population fixture (tests/test_m6_population.py) is rebuilt to the real
seven-column demographic schema plus a real-schema death-records sibling, so this
defect class cannot re-hide behind a flattering fixture.

Also in this PR

• Docs amendment 3f (§2.8.3f): corrects §2.8.3's per-field list, which
  attributed sex to panels.demographic_panel; records the crash
  (m6_population.py:139, grading 4972045579), the canonical source, and the fix;
  adds the revision-log entry and updates the amendment-history JSON block
  (amendment_3f_section + next).
• Sidecar provenance: contract.environment_block now records a
  fitting_stack block (populace-fit / populace-frame version + git rev when
  importable, "absent" otherwise) — the frame-stack vintage the sidecar omitted.
• Runner env docs: run_gate_m6_candidate1.py now names the fitting-stack
  prerequisites (populace-fit in a dedicated venv; the W1 two-env split;
  policyengine-us==1.752.2 + POPULACE_DYNAMICS_PE_US_DIR).
• Stale guard: 4971244215 added to _KNOWN_STALE_REGISTRATIONS and the
  runner-test parametrize.
• Tier recount: tier_counts.json unit 420→427, artifact 1011→1012,
  integration_psid 800→802 (CI-equivalent pytest tests/ -k tier_policy).

Does not touch gates.yaml, runs/, or any floor. Do not merge — an
adversarial referee round (including real-frame population construction) follows.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
social-security-model	Ready	Preview, Comment	Jul 14, 2026 7:23pm

[MaxGhenis]

Adversarial referee round — PR #203 (M6 sex-source patch, pre-fourth-registration)

Scope per grading #42 comment 4972045579 remedy step 3: every PR claim re-derived independently; real-frame population construction executed (loaders → build_realized_population only — no pre-flights, no scoring, no artifact writes). Referee worktree at head ec3e3a3, disposable venv (py 3.14.4 / numpy 2.5.1 / pandas 3.0.3 — matching the graded run env), staged PSID.

The load-bearing claim: one canonical sex source, patch-legal, no design choice — SURVIVED

I hunted for a competing source across every sex read in src/ (data/, engine/, harness/, models/, the deployment-frame transport code, uk/). Person sex for PSID persons originates in exactly two raw variables, and the origination map is closed (no other read_psid column list touches a sex variable):

• ER32000 (deaths.read_death_records, one row per individual, person_id unique on the real file — verified). Every certified attach of person-constant sex onto the demographic-panel/status universe uses it: household_composition.join_demographics (:400, sourced at :446-449), disability.attach_sex (:443-446), and m6_cells.mortality_slices — all three verified at the cited lines.
• MH4 (marriage.marriage_history). It only ever travels with marriage-file-derived rows: transitions.person_attributes, couple_earnings.build_couple_panel (which loads death records but deliberately takes sex from the marriage file), and the gate-2c couple axis (refit.py:892). No certified path attaches MH4 sex to demographic-panel rows. (births.py CAH5/CAH12 are event attributes parent_sex/child_sex; the deployment frame's is_female is the transport universe; uk/ is UKHLS.)

So for the join this patch needed — sex onto anchor persons drawn from the demographic panel — there was exactly one certified pattern, and the patch mirrors it. Patch-legal; no design choice. Bonus evidence: the pre-patch §2.8.3 already contradicted itself (demo → sex at ~:998 AND data.deaths → sex "for the mortality slices" at :1007) — 3f resolves the contradiction in the only direction the code ever supported. Empirically, MH4 and ER32000 agree for 23,695 of 23,696 coded anchor∩marriage-file persons (see N1).

Real-frame proof — REPRODUCED BYTE-EXACT

quantity	PR claim	referee
demographic_panel	892,639 × 7, no sex	892,639 × 7, no sex ✓
death_records	85,536	85,536 ✓ (person_id unique; codes 1: 42,384 / 2: 43,151 / 9: 1 — matches the codebook note incl. 49.55% male)
anchor persons	29,792	29,792 ✓
initial slice / openers	23,094 / 4,571 / 2,127	23,094 / 4,571 (2017) / 2,127 (2019) ✓
seed sex coverage	100%, female 15,499 / male 14,293, 0 nulls	identical ✓

The single ER32000-NA record is not an anchor person, so the full-coverage invariant holds with zero slack on this vintage — if a re-staged vintage ever pulled that person into the anchor, the run raises rather than silently altering the universe. Correct one-shot behavior.

Join mechanics — verified adversarially

All probes pass: agreeing duplicate rows collapse; conflicting coded values raise; NA-only persons are excluded from the map and caught by the anchor-coverage raise; missing person_id/sex columns raise; dtype-mismatched keys fail loud (all-NaN → anchor raise). The runner passes inputs.death_records (m6_runner.py:360); M6HarnessInputs.death_records pre-exists; the new parameter is required-keyword, and the only production call site is the runner (nothing else consumes the signature). Note the deliberate strictness divergence: certified joins drop uncoded persons, the seed join raises for uncoded anchor persons — disclosed in the PR and the right choice for a one-shot (see N3).

Schema audit — discriminating, and complete over its declared surface (probe-proven)

• Unit + psid tiers pass here (psid tier skips on CI — no staged PSID there — so my local run is the live check: 2 passed in 59.97s, including the exact committed-vs-real drift test in both directions).
• The discriminating test discriminates from the schema side; I additionally mutated the manifest side (phantom sex read from demographic_panel) — flagged: build_realized_population reads ['sex'] absent from 'demographic_panel'.
• Completeness probe (mechanical): I stripped all 9 real frames to exactly the manifest's union of reads and re-ran all 8 phases on real data — any unlisted read would crash. Everything completed with byte-identical counts (anchor 29,792; initial 23,094; openers 4,571/2,127; sex 15,499/14,293/0 nulls). No audited phase reads outside the manifest.
• Post-population scored-path phases: pre-flight 1's raw-frame contact is prepare_m6_preflight_context = the same _truncate_household_context the run-2-executed refit used (real-frame-exercised); the projection-time panel builders' reads are within the drift-pinned committed schemas — except marital.attrs, see S2.

Everything else

• 3f amendment: §2.8.3 per-field correction accurate; 2.8.3f cites the grading comment and the crash lines correctly (m6_population.py:139/:186, m6_runner.py:345/:355 — all match the graded forensics); the m6-design-parameters JSON parses, amendment_3f_section present, next lineage extended consistently (third registration failure → 3f → referee → fourth). Revision log updated — but see S1.
• Sidecar fitting_stack: JSON-safe absent ("absent"/"absent") and present ({version, git_rev} shape verified; editable-style resolution returns the correct checkout HEAD). One corner found — S3.
• Runner env docs, stale guard: prerequisites match the graded run-1 forensics; 4971244215 rejected in both bare and URL form (the rsplit("-", 1) normalization covers issuecomment- URLs); parametrize extended.
• Fixtures: the population fixture is exactly the real 7-column demo schema + a faithful read_death_records sibling (dtypes included); the two new tests pin sex-from-sibling and the uncoded-anchor raise.
• Tier recount: pytest tests/ -k tier_policy → 1 passed against actual collection (427/1012/802; +7 unit = 5 audit + 2 population, +2 psid = audit psid tier, +1 artifact = the new stale-guard parametrize case). black --check and ruff check clean on all changed files.
• Test evidence, stated precisely: CI at head runs the full suite minus the psid-gated tests (green, 3.11 + 3.13). Locally I ran every psid-gated test that touches changed code — the two audit tests (green) — plus all 31 tests in the four changed test files and the tier recount (green). Grep-verified that no other psid/reproduction-tier test imports any changed module, so the local full-suite run (still grinding through the unchanged-surface psid tier at post time) carries no additional information about this diff.
• Scope: 8 commits map 1:1 onto the graded remedy list; git diff --name-only confirms no gates.yaml, no runs/, no floor contact. The reconcile commit (05e72cd) widened committed schemas toward reality (disability_status + sequence/status; marital.events + origin) — the honest direction, enforced thereafter by the exact drift test.
• CI: fresh at head ec3e3a3 — pytest 3.11 ✓, 3.13 ✓, lint ✓, build ✓; mergeable: MERGEABLE, mergeStateStatus: CLEAN.

---

Findings

BLOCKING

None.

SHOULD-FIX

S1 — §10 revision counter not bumped for 3f. The doc's own convention ties each amendment to a §10 revision (3d → "§10 revision 10", 3e → "§10 revision 11", both bumped in their landing PRs — master's JSON already carries "revision": 11 from 3e). 3f adds amendment_3f_section but leaves "revision": 11, and its revision-log entry cites no §10 revision. The fourth registration will cite this doc's lineage; bump to 12 (and add "§10 revision 12" to the log entry) or state in the entry why 3f is not a revision event.

S2 — extend the audit to the projection-time native-panel surface (marital.attrs + panel-builder reads). engine/panel_builders.marital_panel_builder (scored-path, projection phase, never yet executed on real frames) reads marital.attrs (person_id, censor_year, start_exposure_year, weight) — a frame absent from both the manifest and COMMITTED_FRAME_SCHEMAS, so the audit does not see that contract. household_panel_builder's support/state/transition reads (age, band, sex, weight, has_next, next_*) are likewise outside the manifest, though pinned transitively by the exact drift test. Why not blocking: I verified the real attrs (9 columns) satisfies every read; person_attributes' output schema is code-determined (fixed .agg), so the run-2 defect class (data-determined schema vs flattering fixture) cannot arise there without code drift; household_panel_builder self-validates with an explicit raise; and the panel-builder unit fixture hand-mirrors the real 9-column attrs schema exactly. Cheap closure before the fourth registration: add marital.attrs to COMMITTED_FRAME_SCHEMAS + a marital_panel_builder/household_panel_builder manifest entry (the psid tier already builds the panel it would check).

S3 — _package_git_revision can attribute the enclosing repo's HEAD to a wheel install. Demonstrated in the referee venv: _package_git_revision("numpy") (wheel in .venv inside the worktree) returns the worktree's HEAD (ec3e3a3…), not "unknown" — git rev-parse succeeds from site-packages because the venv lives inside a git repo, which is exactly the documented .venv-wt-in-repo layout. With the documented editable populace-fit install the recorded rev is correct (verified: source-tree resolution returns the right checkout HEAD), but a non-editable deviation would write wrong-but-plausible provenance into a certification sidecar — worse than "unknown". Cheap guard: return "unknown" when the resolved location sits under site-packages/dist-packages, or verify the git root owns the distribution.

S4 — test hardening (three small gaps). (a) _person_sex_map's conflicting-coded-values raise has no test (I verified the behavior manually; a future "simplification" of the dedup could silently turn conflicts into first-wins). (b) The audit's phase list isn't pinned — deleting a PHASE_FRAME_COLUMN_READS entry silently narrows the audit with all tests green; assert the expected 8 phase names. (c) Nothing ties the population fixture's demo columns to COMMITTED_FRAME_SCHEMAS["demographic_panel"] — assert equality so the fixture can't silently re-flatter.

NOTE

N1 — MH4 vs ER32000 disagree for exactly 1 of 23,696 coded anchor∩marriage-file persons. That person is classified by MH4 sex in the marital truth cells (marital_tables_from_panel carries panel sex) and by ER32000 in the seed/mortality/disability. Pre-existing certified behavior, not introduced or worsened here; immaterial at cell granularity; recorded so the subsystem split is on the record.

N2 — wording in 2.8.3f: "the same canonical attach the certified deployment builders use" — join_demographics/attach_sex are the certified gate-2 builders; "deployment" collides with the deployment-frame transport code, which sources sex differently (is_female, different universe). Suggest "certified builders" (the JSON block already says it correctly).

N3 — raise-vs-drop divergence, endorsed: certified joins drop uncoded persons; the seed join raises for uncoded anchor persons. Dropping would silently shrink the seed universe (a design choice); raising forces any such vintage change to a human. Correct for a one-shot, and disclosed.

---

Verdict

The core fix is correct and canonical, the proof reproduces byte-exact, the audit is discriminating and probe-complete over its declared surface, scope is clean, CI is fresh and green at head. The amend list is small and none of it touches the sex-source logic: S1 is a two-line docs-lineage fix that the fourth registration's citations should not inherit wrong; S2–S4 are cheap hardening that belongs in this PR's spirit (the audit exists to make the next crash class impossible, and S2/S4b are the two ways it can silently under-deliver).

VERDICT: AMEND — land S1 (mandatory) and S2–S4 (strongly recommended, ~1 hour together), then this is merge-ready without a further referee round; re-run pytest tests/ -k tier_policy after S4 and the psid audit tier after S2.

[MaxGhenis]

Referee AMEND applied (comment 4972996254)

All four SHOULD-FIX landed on this branch, one commit each; core sex-source logic untouched.

#	fix	commit
S1	bump JSON "revision" 11→12 and cite "§10 revision 12" in the 3f revision-log entry (3d/3e convention)	b5516f2
S2	extend the audit to the projection-time native-panel surface: add marital.attrs (9-col person_attributes schema) to COMMITTED_FRAME_SCHEMAS + marital_panel_builder / household_panel_builder manifest entries; marital.attrs now rides the phantom-read discrimination and the exact psid drift check	323d75c
S3	_package_git_revision returns "unknown" for a wheel resolving under site-packages/dist-packages (no more enclosing-repo-HEAD misattribution) + numpy→"unknown" regression test	84b7ff0
S4	(a) _person_sex_map conflicting-coded-values raise test; (b) pin the expected 10-phase manifest set; (c) assert the demo fixture equals COMMITTED_FRAME_SCHEMAS["demographic_panel"]	4219f31

N2 (wording) folded into S1: §2.8.3f now says "certified builders", not "deployment builders". N1 (1-of-23,696 MH4-vs-ER32000 disagreement — pre-existing, immaterial at cell granularity) and N3 (raise-vs-drop divergence — endorsed) need no action, per your review.

Tier recount after the new tests: unit 427→431 (3243cee); artifact/integration_psid unchanged. psid audit tier green locally (marital.attrs drift check exact in both directions); black/ruff clean. Not merged — awaiting your re-confirmation.

[MaxGhenis]

Referee verification round — fixes re-derived at head 3243cee

Fresh worktree at head, disposable venv, staged PSID. Each of my four SHOULD-FIX re-checked mechanically (guard/pin mutated → must fail), not read for a "fixed" label. The four fix commits (b5516f2/323d75c/84b7ff0/4219f31) stack cleanly on my reviewed head ec3e3a3, so provenance is exact.

Core sex-source logic untouched since review: git diff ec3e3a3..3243cee on m6_population.py and m6_runner.py is empty — the _person_sex_map join, the coverage invariant, and the runner call are byte-identical to what I reviewed.

Per-finding

S1 — RESOLVED. JSON m6-design-parameters parses; "revision": 11 → 12; the revision-log entry now cites "§2.8.3f, §10 revision 12" (matches the 3d→rev-10 / 3e→rev-11 convention). N2 folded in: §2.8.3f now reads "the certified builders use", not "deployment builders".

S2 — RESOLVED. marital.attrs (the 9-column person_attributes schema) added to COMMITTED_FRAME_SCHEMAS, plus marital_panel_builder + household_panel_builder manifest entries (audited phases 8 → 10). It rides both guards, verified live: (a) phantom-read discrimination — I injected a bogus read into marital_panel_builder["marital.attrs"] and the audit flagged it (reads ['phantom_col'] absent from 'marital.attrs'); (b) exact psid drift check — marital.attrs is now in _real_frames(), and test_committed_schemas_match_the_real_loader_columns passes, so committed == real in both directions on staged PSID. The two new projection-phase entries are satisfiable against the real loaders (test_manifest_reads_are_satisfied_by_the_real_loaders green) — the never-run-on-real-frames surface I flagged is now pinned.

S3 — RESOLVED. The site-packages/dist-packages guard fixes the misattribution I demonstrated: on this build _package_git_revision("numpy") returns "unknown" (round 1 it returned the enclosing worktree's HEAD). The guard is surgical — the legitimate editable case still resolves (_package_git_revision("populace_dynamics") == HEAD 3243cee…). The regression test discriminates: it asserts numpy → "unknown", which fails against the pre-fix code.

S4 — RESOLVED. (a) test_person_sex_map_raises_on_conflicting_coded_values — present, green, matches on "conflicting"; (b) test_manifest_pins_the_expected_phase_set pins exactly the 10 phase names — deleting any entry flips the set inequality (verified); (c) test_demo_fixture_matches_the_committed_demographic_schema asserts the fixture columns equal COMMITTED_FRAME_SCHEMAS["demographic_panel"] (no sex), closing the silent-re-flatter path.

Gate checks

• 24/24 in the four changed test files green, incl. the psid audit tier (marital.attrs drift check exact both directions, 82.7s).
• Tier recount CI-equivalent: pytest -k tier_policy passes against actual collection; counts {unit 431, artifact 1012, integration_psid 802, reproduction_legacy 520, oracle_policyengine 159} = 2,924 total (unit 427 → 431 = the 4 new unit tests; artifact/psid unchanged).
• black --check + ruff check clean on all 6 changed files.
• Fresh CI at head 3243cee: pytest 3.11 ✓, 3.13 ✓, lint ✓, build ✓; mergeable: MERGEABLE, mergeStateStatus: CLEAN.

VERDICT: VERIFIED (ready to merge). All four findings resolved and independently discriminating; N2 folded in; core sex-source logic untouched. No further referee round required.