Past-cap census as a first-class calibration diagnostic - #494
Conversation
…able A) Under the capped weighted-MAPE objective, a row whose scaled miss reaches target_loss_cap carries zero gradient: the solver can neither improve it nor be charged for making it worse. The published Build N sparse run measured the consequence — 17 rows that started inside the cap were pushed past it and abandoned (the free-dumping-ground dynamic on #492) — and nothing reported which rows the solve wrote off. diagnostics_payload now ships a past_cap_census block: initial/final past-cap counts, escaped (past at init, recovered), frozen (past at both ends), pushed_out (in-cap at init, past at final), and the pushed-out row list with each row's scaled misses, worst first. The census reads the cap from the result options (both the calibrate shape and the score_targets shape) and uses the loss's own scale rule max(|target|, 1). Pure observability — no solver behavior change. Schema version 4 -> 5. Refs #492. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner-gate finding on this PR: populace-calibrate here writes schema_version 5 (the past_cap_census block) while populace-data's contract pinned 4 and hard-rejects any other value — the next release build would have passed its gates and then publish-failed on schema version, the exact cross-gate disease #491 closed within the builder. The two same-named constants cannot share code (populace-data must not import populace-calibrate), so the builder suite — the one place that imports both — now pins them equal (test_calibration_diagnostics_schema_lockstep). Contract + release/contract test fixtures move to 5. Older shipped releases (schema <= 4) remain valid history; the contract's version pin has always meant 'what this library version publishes', per its docstring. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Owner gate: found and fixed a cross-package schema break — this PR moved Pushed the lockstep commit: contract + fixtures to 5, plus |
Sol confirm finding: the census used >= cap, classifying a row exactly
AT the cap as zero-gradient. torch.clamp keeps gradient at the boundary
(verified: d/dx clamp(x, max=cap) at x == cap is 1.0), so an at-cap row
still pulls in the objective. 'Past cap' now means strictly greater,
matching the census's own definition ('zero gradient'), and the test
pins the boundary row as alive.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ne solve, validated sidecar (#625) Downstream-only layer over the ACS local artifact — no national-calibration diffs anywhere: - sld_membership: 2024 SLD BEF membership at the ACS target vintage (the artifact's baked sldl/sldu are 2020-BAF, donor-only). One seeded operator: exact/within-tract lookup for tract-carrying rows, block-overlap draws conditional on (PUMA, CD, county) with a recorded fallback ladder for ACS-spine rows; both chambers read off one draw. Conservation-checked overlap tables built by tools/build_us_sld_membership_ladder_artifact.py from pinned sources (NationalSLD*24 BEFs + PL 94-171 + tract-PUMA + cd119). - sld_local_targets: ledger SLD facts (S0101 age bands, B19001 brackets + households, B19013 medians) compiled to per-district problems; medians are validation-only by construction; the ACS money-income analog is a declared recipe over artifact inputs with declared omissions (SSI, TANF) and exclusions. - sld_local_solver + sld_local_doctrine: independent per-district solves reusing populace.calibrate.solve internals; declared cap 10.0 and stretch 100.0 vs the artifact-weight anchor (#493 stated); structurally knob-free doctrine wrappers (#492/#503 ported); per-district past-cap census with chamber roll-ups (#494 generalized). - sld_local_report: achieved-vs-target table, published-median validation, statewide coherence (reported, never constrained), and the honest-boundaries statement generated into the sidecar. - tools/build_us_sld_local_layer.py: membership -> targets -> doctrine solve -> hashed sidecar chain; reads packaged variable/year and pandas-HDF (table + fixed) artifact layouts. - experiments/sld_local_pilot: first real run on the published #512 rebuild (Utah: all 104 districts, median within-10% = 1.00 both chambers, zero frozen/pushed-out rows, median-income sanity median gap 3.3%). The binding 100x ratio and thin-district ESS are recorded for the candidate review. 46 new tests; ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
What
Deliverable A of #492: the past-cap census ships in
calibration_diagnostics.json(and every artifact written throughwrite_calibration_diagnostics, including the read-only scorer). Pure observability — no solver behavior change.New
past_cap_censusblock indiagnostics_payload(schema 4 → 5):initial_past_cap/final_past_cap— rows at or pasttarget_loss_capunder the initial / final estimatesescaped— past at initialization, back inside at finalfrozen— past at both endspushed_out+pushed_out_rows— rows that started inside the cap and were pushed past it by the solve (name,init_rel,final_rel), worst final miss firstMisses are scaled with the loss's own rule
max(|target|, 1), so "past cap" here means exactly "zero gradient in the objective". The cap is read from result options in both recorded shapes (calibrate'starget_loss_scales.capandscore_targets' top-leveltarget_loss_cap); results with no recorded cap census asnull.Why
The measured Build N census (#492): 929 rows past cap at init, 377 at final, and 17 rows pushed out — in-cap rows abandoned past the cap because the 101st percentage point of error is free (LA TANF 0.47 → 5.54; four of the six broken medical states got worse while past cap). Nothing reported which rows the solve wrote off. Now the release artifact does.
Scope
Diff is entirely inside
packages/populace-calibrate(PR #491 owns the builder/populace-data files; no overlap). The builder andtools/score_us_fiscal_targets.pypick the census up for free throughdiagnostics_payload.Testing
max(|target|,1)scale basis (a 0.5-valued target censuses on scale 1, not raw relative error), the ≥-boundary (a row exactly at the cap counts as past), and the worst-first orderingcalibrate(frozen hopeless row),score_targets(score-shape cap pickup; no motion ⇒escaped == pushed_out == 0), andcalibrate_l0_refit(merged-options cap)initial_past_cap == escaped + frozen,final_past_cap == frozen + pushed_outruff format+ruff checkcleanRefs #492.
🤖 Generated with Claude Code