DRAFT: SIPP-vs-J2J seam-reconciliation artifact (pre-C3) - #214
Closed
daphnehanse11 wants to merge 3 commits into
Closed
DRAFT: SIPP-vs-J2J seam-reconciliation artifact (pre-C3)#214daphnehanse11 wants to merge 3 commits into
daphnehanse11 wants to merge 3 commits into
Conversation
…192 protocol) The reconciliation run the protocol rules require before E2/E4 thresholds lock, now with the E->N leg the posted groundwork lacked (person-month universe presence, so exits to nonemployment count). Within-wave monthly job separation: 1.71% (ref 2021) / 1.82% (ref 2022). Across-wave Dec->Jan (10,051 persons linked on wave- consistent job ids): 9.45% — the seam runs ~5x within-wave on the identical definition. J2J national monthly-equivalent (~4.1%) sits near the geometric midpoint: within-wave understates ~2.3x, the seam overstates ~2.3x. Concept deltas named in the artifact; the proposed ruling (J2J = rate levels, SIPP = persistence structure, hazards seam-aware) is recorded NOT RATIFIED for the C3 round. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
… naming note Metadata/caveat-only changes; no measured numbers touched. Artifact hand-edited (SIPP microdata not staged here); script updated to emit the same fields on regeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
daphnehanse11
marked this pull request as ready for review
July 16, 2026 14:30
Brings the tier manifest and test suite up to master so collected tier counts match tests/tier_counts.json (unit 651). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vahid-ahmadi
left a comment
vahid-ahmadi
left a comment
Contributor
There was a problem hiding this comment.
DRAFT-appropriate feedback. This is unusually candid — naming the concept deltas up front, and flagging its own most dangerous assumption rather than burying it, is the right instinct. Comments split by whether they block merging (few) versus block the number being used to set C3 thresholds (several).
Verified as correct on my read: the J2J aggregation sums firmsize 1–5 with industry=="00" and the extract has no all-firmsize total row, so no double counting; range(1, 12) correctly yields 11 month-pairs; the file-year→reference-year mapping lines pu2022/pu2023 up with the J2J 2021/2022 window.
Comparability — these block threshold use
1. The SIPP side is entirely unweighted. WPFINWGT is never read; person_month_presence pulls only SSUID/PNUM/MONTHCODE. J2J's MainB/MSep are population counts. So a weighted population rate is being differenced against an unweighted sample rate. SIPP oversamples low-income households, which separate more — biasing the SIPP side upward, the same direction as the seam effect being isolated. This isn't in concept_deltas and should be.
2. No age or universe restriction on the SIPP side. J2J covers UI-covered workers; the SIPP universe here is every person-month with a job record, all ages. Youth jobs are high-turnover. The delta list names UI coverage and self-employment but not the age universe.
3. Self-employment is named but not excluded ("JBORSE 2/3 excludable in later cuts"), so it's a live wedge in the headline number rather than a resolved one.
4. Possibly the wrong benchmark series. The script uses MSep/MainB — all main-job separations. The committed extract also carries J2JSep, EESep, ENSep. For an artifact framed on job-to-job flows, J2JSep, or reporting MSep/EESep/ENSep side by side to decompose the gap, seems more apt — and would remove the need for the ~10× caveat against #212's person-level measure.
5. J2J is treated as ground truth by construction
concept_deltas lists no J2J limitation at all — nothing on self-employment/informal exclusion, the federal-employment gap, or that J2J firm-to-firm transitions are themselves partly imputed from EIN/SEIN successor logic. The list runs one-directionally against SIPP.
Separately, the "J2J sits near the geometric midpoint (within-wave understates ~2.3×, the seam overstates ~2.3×)" observation in the PR body is a symmetry noticed after computing the numbers, and it's doing rhetorical work in support of the ruling. It appears nowhere in the artifact JSON.
If the ex-ante rule ("J2J is truth for LEVELS, SIPP for persistence STRUCTURE") genuinely exists in #192, please quote it verbatim in the artifact so a referee doesn't have to take it on trust. If it doesn't, label it as judgment-after-the-number — which is fine for a DRAFT anchor, just not as a pre-specified rule.
6. The identification rests on the assumption #235 was meant to verify
concept_deltas[4] flags as UNVERIFIED that SIPP EJB job IDs are longitudinally consistent across the pu2022→pu2023 boundary. If IDs are reassigned at the file boundary, the entire 5× seam-to-within-wave contrast is a linkage artifact. Credit for flagging it rather than burying it.
Worth being explicit about the dependency structure, though: the 9.45% here is the identical computation to #235's, on the same 10,051 linked persons / 10,828 jobs. That's agreement, not independent corroboration — both inherit the same assumption. And #235's own PASS has issues I've raised there (the pre-registration label doesn't match the commit record, and the verdict flips to PASS_WITH_CORRECTION_BAND if scored on the E→E population). So I'd avoid treating #235 as having discharged this caveat until those are settled.
Non-blocking
- J2J extract has no provenance sidecar. Committed in 80f55ae (#195) with no vintage, pull date, or checksum, while every SSA source in
data/external/carries.source.provenance.json. J2J revises across releases, so a rerun silently changes the benchmark. Worth matching the existing convention. - SIPP inputs unpinned (env var, no checksum/version assertion). Deterministic, so no seed needed.
- No test pins the artifact. Fine for DRAFT; needed before C3.
- Within-wave mean averages 11 monthly rates rather than pooling numerators and denominators. Denominators are close enough that it barely moves, but pooling is cleaner.
- Quarterly→monthly
1-(1-q)^(1/3)is correctly flagged as an approximation. Note it's biased downward against a hazard with duration dependence (separations concentrate early in a spell), so ~4.1% is a soft lower bound. Doesn't change the ordering of the three numbers.
Bottom line
Reasonable to merge as a reported anchor. Items 1–5 should block it being used to set C3 thresholds, and item 6 is the one I'd resolve first — the cheap job-ID verification is carrying more weight than its current state supports.
Collaborator
Author
|
This PR was accidentally closed when its head branch was deleted during a cleanup of merged branches (my mistake — the branch has been restored at the identical head |
vahid-ahmadi
added a commit
that referenced
this pull request
Aug 11, 2026
) * DRAFT: SIPP-vs-J2J seam-reconciliation artifact (pre-C3, required by #192 protocol) The reconciliation run the protocol rules require before E2/E4 thresholds lock, now with the E->N leg the posted groundwork lacked (person-month universe presence, so exits to nonemployment count). Within-wave monthly job separation: 1.71% (ref 2021) / 1.82% (ref 2022). Across-wave Dec->Jan (10,051 persons linked on wave- consistent job ids): 9.45% — the seam runs ~5x within-wave on the identical definition. J2J national monthly-equivalent (~4.1%) sits near the geometric midpoint: within-wave understates ~2.3x, the seam overstates ~2.3x. Concept deltas named in the artifact; the proposed ruling (J2J = rate levels, SIPP = persistence structure, hazards seam-aware) is recorded NOT RATIFIED for the C3 round. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review: job-ID linkage caveat, year-label convention, j2j/J2J naming note Metadata/caveat-only changes; no measured numbers touched. Artifact hand-edited (SIPP microdata not staged here); script updated to emit the same fields on regeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Vahid Ahmadi <va.vahidahmadi@gmail.com>
vahid-ahmadi
added a commit
that referenced
this pull request
Aug 11, 2026
… (pre-lock, #230 §6) (#235) * Cross-wave job-ID consistency check: PASS (pre-lock artifact for #230 §6) The blocking check for the seam ruling (ADR 0004 referee item 7; #214 concept-delta 5). Verdict rule pre-registered in the script before the numbers were seen. Results: gross ID survival across the pu2022->pu2023 boundary 90.55%; re-key signature (same industry + class of worker + earnings within 20%) among seam separators-to- employment 17.5% vs a 2.4% within-wave coincidence baseline; scaled to all seam separations (38.1% are exits to nonemployment, which cannot be ID artifacts), the implied ID-artifact share of the 9.45% seam rate is <= 9.4% — under the 15% PASS bar. At least ~90% of the seam contrast is real seam-bunched separation; the #214 ruling's conditional check is satisfied. Implementation note, disclosed: the first run inner-joined the next-month jobs frame and silently dropped exits to nonemployment (printing a 6.06% conditioned seam rate and a PASS_WITH_CORRECTION_BAND verdict against the wrong denominator); the fix restores the documented design via the person-month universe and reproduces #214's 9.45%/1.77% exactly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Correct pre-registration label; report both scoring populations (review of #235) Addresses the blocking items in the #235 review. No measured number changed; no computation altered. - Withdraw the "pre-registered" claim. Rule and result land in one commit (87788eb) with no prior threshold on record, and a first run returned a different verdict before the estimator was corrected. Relabelled as disclosed re-analysis after a discovered defect. - Publish both scoring populations. E->E excess is 0.1512 (PASS_WITH_CORRECTION_BAND); scaled by the E->E share it is 0.0936 (PASS). Derivable from counts already in the artifact. Which is operative is left OPEN for the C3 referee round -- deliberately not chosen here, since choosing after seeing both sides of the bar is the defect this file documents. - Relabel gross_id_survival_share as an identity (1 - sep_rate), not a bound; drop the dead jobs_kept_share branch. - Record the 15/30 bands as having no derivation, pending ratification. - Register known biases: NaN-as-agreement in _rekey_match, unmotivated EARN_LOG_TOL, baseline composition mismatch, and the seam-denominator circularity in person presence. Artifact edited to match without re-running (SIPP microdata not on this machine); edit note records that every added value is recomputable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review: disclosed re-analysis, both populations, pinning All items from the changes-requested review: - Pre-registration language corrected everywhere to DISCLOSED RE-ANALYSIS with the full first-run history in the status field; the 15/30 bands and the operative scoring population are marked UNRATIFIED referee items. - Both populations reported with the verdict split visible: E->E 15.12% (PASS_WITH_CORRECTION_BAND) vs all-separations 9.36% (PASS); operative = REFEREE. - One-sided 95% uppers propagated from binomial SEs (17.65% / 10.92%) replace the bare point estimate. - gross_id_survival relabelled as a definitional identity with the dead branch removed. - Strict-NaN sensitivity variant added (missing fields = mismatch): excess 12.32% / 7.62% — the NaN-matching bias was real and its correction lowers the estimate. - Composition-mismatch and seam-denominator (person-ID linkage under test) caveats recorded. - Inputs sha256-pinned; artifact-tier test pins the disclosure language, both populations, the identity label, uncertainty, the strict variant, and the pins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Referee S3 mechanics: recompute test, E->E-conditional definition, reader pin Per the #230 round-1 referee review: derived fields and both verdicts now recompute from the committed counts in an artifact-tier test (a hand-edited verdict fails); rekey_signature_definition states the E->E-conditional baseline the script actually computes (the per-all-seps share fields are marked descriptive); the sipp_jobs reader commit is pinned in the artifact alongside the input sha256s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Drop the stray .claude gitlinks my review commit added; ignore .claude My `633ad10` on this branch accidentally staged eleven `.claude/ worktrees/agent-*` entries — local Claude Code worktrees, committed as gitlinks (mode 160000) to commits that exist in no remote. They are unrelated to this PR and would land on master as broken submodule references that `git clone` cannot resolve. Removed from the index (the local directories are untouched) and `.claude/` added to `.gitignore` so the mistake cannot recur on any branch. My error, cleaned up on the branch it landed on rather than left for the artifact's author. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Recount tier manifest after merging master (artifact 2,528 -> 2,535) The seven cross-wave job-ID check tests read committed runs/*.json evidence and therefore classify into the artifact tier, not unit. Counts recollected from the committed tree, per the merge convention. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Vahid Ahmadi <va.vahidahmadi@gmail.com>
vahid-ahmadi
added a commit
that referenced
this pull request
Aug 11, 2026
…236) * Document C3 precision-first linkage QC Co-Authored-By: Codex gpt-5.6-sol <noreply@openai.com> * Wire linkage QC into the employer battery Co-Authored-By: Codex gpt-5.6-sol <noreply@openai.com> * Cross-wave job-ID consistency check: PASS (pre-lock artifact for #230 §6) The blocking check for the seam ruling (ADR 0004 referee item 7; #214 concept-delta 5). Verdict rule pre-registered in the script before the numbers were seen. Results: gross ID survival across the pu2022->pu2023 boundary 90.55%; re-key signature (same industry + class of worker + earnings within 20%) among seam separators-to- employment 17.5% vs a 2.4% within-wave coincidence baseline; scaled to all seam separations (38.1% are exits to nonemployment, which cannot be ID artifacts), the implied ID-artifact share of the 9.45% seam rate is <= 9.4% — under the 15% PASS bar. At least ~90% of the seam contrast is real seam-bunched separation; the #214 ruling's conditional check is satisfied. Implementation note, disclosed: the first run inner-joined the next-month jobs frame and silently dropped exits to nonemployment (printing a 6.06% conditioned seam rate and a PASS_WITH_CORRECTION_BAND verdict against the wrong denominator); the fix restores the documented design via the person-month universe and reproduces #214's 9.45%/1.77% exactly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Correct pre-registration label; report both scoring populations (review of #235) Addresses the blocking items in the #235 review. No measured number changed; no computation altered. - Withdraw the "pre-registered" claim. Rule and result land in one commit (87788eb) with no prior threshold on record, and a first run returned a different verdict before the estimator was corrected. Relabelled as disclosed re-analysis after a discovered defect. - Publish both scoring populations. E->E excess is 0.1512 (PASS_WITH_CORRECTION_BAND); scaled by the E->E share it is 0.0936 (PASS). Derivable from counts already in the artifact. Which is operative is left OPEN for the C3 referee round -- deliberately not chosen here, since choosing after seeing both sides of the bar is the defect this file documents. - Relabel gross_id_survival_share as an identity (1 - sep_rate), not a bound; drop the dead jobs_kept_share branch. - Record the 15/30 bands as having no derivation, pending ratification. - Register known biases: NaN-as-agreement in _rekey_match, unmotivated EARN_LOG_TOL, baseline composition mismatch, and the seam-denominator circularity in person presence. Artifact edited to match without re-running (SIPP microdata not on this machine); edit note records that every added value is recomputable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review: disclosed re-analysis, both populations, pinning All items from the changes-requested review: - Pre-registration language corrected everywhere to DISCLOSED RE-ANALYSIS with the full first-run history in the status field; the 15/30 bands and the operative scoring population are marked UNRATIFIED referee items. - Both populations reported with the verdict split visible: E->E 15.12% (PASS_WITH_CORRECTION_BAND) vs all-separations 9.36% (PASS); operative = REFEREE. - One-sided 95% uppers propagated from binomial SEs (17.65% / 10.92%) replace the bare point estimate. - gross_id_survival relabelled as a definitional identity with the dead branch removed. - Strict-NaN sensitivity variant added (missing fields = mismatch): excess 12.32% / 7.62% — the NaN-matching bias was real and its correction lowers the estimate. - Composition-mismatch and seam-denominator (person-ID linkage under test) caveats recorded. - Inputs sha256-pinned; artifact-tier test pins the disclosure language, both populations, the identity label, uncertainty, the strict variant, and the pins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Referee S3 mechanics: recompute test, E->E-conditional definition, reader pin Per the #230 round-1 referee review: derived fields and both verdicts now recompute from the committed counts in an artifact-tier test (a hand-edited verdict fails); rekey_signature_definition states the E->E-conditional baseline the script actually computes (the per-all-seps share fields are marked descriptive); the sipp_jobs reader commit is pinned in the artifact alongside the input sha256s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * E4/E5 minimal-audit manifest: registered design (ADR 0004, pre-lock) Workstream A's #230 section-12.2 pre-lock artifact: the complete ADR 0004 adjudication design for the first-lock scope (SIPP-internal employer attachment). Registers the frame (the #235 population), the two arms (accepted-assignment precision; truth-search recall over the re-key class), scoped stratification with seam oversampling, REFEREE slots for P_floor/P_design/alpha/power with a worked binomial example (0.95/0.99/0.05/0.80 -> n=124, c=122 per stratum before clustering and inflation), the blinded ID-masked coding protocol with conservative indeterminate handling registered before labels, provenance including the draw seed (20260717), and the two leakage freezes (the #235 signature parameters; no label backflow into readers or hazards). Sample draws only after the referee round fills the slots. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Pin the audit-result sequence per referee S4 (ADR 0004 section 1.5) Manifest pre-lock; lock may proceed with the audit designed but undrawn; results must exist before the first one-shot candidate run scoring any E4/E5 cell. Failed floor invalidates the cells (no candidate can pass) and the artifact publishes regardless — a designed stop is a graded, publishable outcome. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Rename the interface contracts C1/C2/C3 -> IC1/IC2/IC3 (ADR 0003 am. 1) Naming only: no column, band, code mapping, or gate definition changes, and the numbering is preserved 1:1 so every prior reference maps by prefixing "I". A bare "C1" meant four different things in this repo: the gate_w1 fingerprints (gates.yaml fingerprints.c1/.c2), the SSA Trustees table II.C1, an RNG substream in household composition, and the employer-firm interface contracts. Only the last is repo-internal, pre-lock, and ours -- the fingerprints sit inside gate_w1, which is locked: true, so renaming those would cost a public amendment plus a fresh referee round. Timing is the point. IC3 (the employer gate block) is about to be written into gates.yaml, which already contains fingerprints.c1 and fingerprints.c2. A block named C3 locking next to them makes either rename cost exactly what the fingerprint row of that table already costs. Vahid flagged the collision on #192 before the referee round; this closes it while it is still free. Touches frozen ADR 0003 text, so it is a joint-PR change under the IC1/IC2 freeze rule -- procedurally, not because anything moved. Prior discussion keeps the old names; the ADR carries the mapping. Also corrects a stale claim in sipp_jobs: the module said "ADR 0003 is Proposed, not frozen" as the reason job_spells is IC1-preview. It is Accepted and IC1 is frozen; what is still preview-grade is the collapse's single-ref_year coverage, which is what the docstring now says. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Add the E5 run arm, no-revisit clause, frame-measured power, registered pooling Responds to the 2026-07-23 review round (items 1-4 of the 2026-07-19 review restated there): 1. E5 scope: new run arm (c) — unit is the maximal same-ID chain; every internal link and both terminal transitions are coded, so run-level error (false continuation / false break) is measured directly rather than composed from pair precision by an independence assumption. Scope mapping registered in section 1: arms (a)/(b) certify E4, arm (c) certifies E5. 2. No-revisit clause: all REFEREE slots, including the arm-(b) gates-or-reported decision, must be filled before the draw and may not be revised after any label exists. 3. Power: worker-resampling simulation from the actual frame, so deff is measured not assumed; arm-(b) target formula registered with the #235 prevalence priors; run-arm budget in link-codings. 4. Pooling rule registered now (fixed collapse order, never across the within-wave/seam axis), not deferred to the draw artifact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review: finish the Proposed fix, rename the operative plan Two should-fixes from Daphne's #277 review: 1. The stale "ADR 0003 (Proposed — this is a preview, not the frozen contract)" text was corrected in the module docstring but survived in `job_spells`'s own docstring. Post-merge the ADR says Accepted while that line said Proposed. It now mirrors the module wording: the schema is frozen; what is preview-grade is this collapse's single-`ref_year` coverage. 2. `docs/plans/employer-firm-plan.html` used C1/C2/C3 in the contract sense while being cited by the ADR's Context section as the operative split — the one file on the wrong side of the rename boundary. Renamed to IC1/IC2/IC3 (six lines; the SVG path data containing `C265,247` is untouched), and Amendment 1 now states the boundary explicitly: history keeps the old names, live documents are renamed, and the three unrenamed senses stay as the table gives them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Drop the stray .claude gitlinks my review commit added; ignore .claude My `633ad10` on this branch accidentally staged eleven `.claude/ worktrees/agent-*` entries — local Claude Code worktrees, committed as gitlinks (mode 160000) to commits that exist in no remote. They are unrelated to this PR and would land on master as broken submodule references that `git clone` cannot resolve. Removed from the index (the local directories are untouched) and `.claude/` added to `.gitignore` so the mistake cannot recur on any branch. My error, cleaned up on the branch it landed on rather than left for the artifact's author. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fold linkage-QC adoption requirements into ADR 0004 * Keep naming amendment outside sealed production sources * Align linkage QC with IC naming and aggregate gate boundary * Close pre-draw audit sizing channels --------- Co-authored-by: Max Ghenis <mghenis@gmail.com> Co-authored-by: Codex gpt-5.6-sol <noreply@openai.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Vahid Ahmadi <va.vahidahmadi@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The reconciliation run the #192 protocol rules require before E2/E4 thresholds lock, committed in the reported-anchor convention (script +
runs/artifact, DRAFT/NOT-RATIFIED). Supersedes the posted groundwork by adding the E→N leg via the person-month universe, so exits to nonemployment count as separations.The seam runs ~5× within-wave on the identical definition, and J2J sits near the geometric midpoint (within-wave understates ~2.3×, the seam overstates ~2.3×). Every remaining concept delta is named in the artifact (job-based counting, UI coverage, main-job vs all-jobs, the quarterly→monthly approximation). The proposed ruling — J2J is truth for rate levels, SIPP for persistence structure, phase-1 hazards estimated seam-aware — is recorded explicitly NOT RATIFIED; it belongs to the C3 referee round, per the plan. Consumes #195's committed J2J extract and #206's reader.
🤖 Generated with Claude Code