Skip to content

Past-cap census as a first-class calibration diagnostic - #494

Merged
MaxGhenis merged 4 commits into
mainfrom
past-cap-census-492
Jul 22, 2026
Merged

Past-cap census as a first-class calibration diagnostic#494
MaxGhenis merged 4 commits into
mainfrom
past-cap-census-492

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

What

Deliverable A of #492: the past-cap census ships in calibration_diagnostics.json (and every artifact written through write_calibration_diagnostics, including the read-only scorer). Pure observability — no solver behavior change.

New past_cap_census block in diagnostics_payload (schema 4 → 5):

  • initial_past_cap / final_past_cap — rows at or past target_loss_cap under the initial / final estimates
  • escaped — past at initialization, back inside at final
  • frozen — past at both ends
  • pushed_out + pushed_out_rows — rows that started inside the cap and were pushed past it by the solve (name, init_rel, final_rel), worst final miss first

Misses are scaled with the loss's own rule max(|target|, 1), so "past cap" here means exactly "zero gradient in the objective". The cap is read from result options in both recorded shapes (calibrate's target_loss_scales.cap and score_targets' top-level target_loss_cap); results with no recorded cap census as null.

Why

The measured Build N census (#492): 929 rows past cap at init, 377 at final, and 17 rows pushed out — in-cap rows abandoned past the cap because the 101st percentage point of error is free (LA TANF 0.47 → 5.54; four of the six broken medical states got worse while past cap). Nothing reported which rows the solve wrote off. Now the release artifact does.

Scope

Diff is entirely inside packages/populace-calibrate (PR #491 owns the builder/populace-data files; no overlap). The builder and tools/score_us_fiscal_targets.py pick the census up for free through diagnostics_payload.

Testing

  • Stub-result test pins the exact classification of every row class, the max(|target|,1) scale basis (a 0.5-valued target censuses on scale 1, not raw relative error), the ≥-boundary (a row exactly at the cap counts as past), and the worst-first ordering
  • Integration tests through calibrate (frozen hopeless row), score_targets (score-shape cap pickup; no motion ⇒ escaped == pushed_out == 0), and calibrate_l0_refit (merged-options cap)
  • Census identities asserted: initial_past_cap == escaped + frozen, final_past_cap == frozen + pushed_out
  • Full populace-calibrate suite green locally; ruff format + ruff check clean

Refs #492.

🤖 Generated with Claude Code

…able A)

Under the capped weighted-MAPE objective, a row whose scaled miss reaches
target_loss_cap carries zero gradient: the solver can neither improve it
nor be charged for making it worse. The published Build N sparse run
measured the consequence — 17 rows that started inside the cap were pushed
past it and abandoned (the free-dumping-ground dynamic on #492) — and
nothing reported which rows the solve wrote off.

diagnostics_payload now ships a past_cap_census block: initial/final
past-cap counts, escaped (past at init, recovered), frozen (past at both
ends), pushed_out (in-cap at init, past at final), and the pushed-out row
list with each row's scaled misses, worst first. The census reads the cap
from the result options (both the calibrate shape and the score_targets
shape) and uses the loss's own scale rule max(|target|, 1). Pure
observability — no solver behavior change. Schema version 4 -> 5.

Refs #492.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner-gate finding on this PR: populace-calibrate here writes
schema_version 5 (the past_cap_census block) while populace-data's
contract pinned 4 and hard-rejects any other value — the next release
build would have passed its gates and then publish-failed on schema
version, the exact cross-gate disease #491 closed within the builder.

The two same-named constants cannot share code (populace-data must not
import populace-calibrate), so the builder suite — the one place that
imports both — now pins them equal (test_calibration_diagnostics_schema_lockstep).
Contract + release/contract test fixtures move to 5. Older shipped
releases (schema <= 4) remain valid history; the contract's version pin
has always meant 'what this library version publishes', per its
docstring.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Owner gate: found and fixed a cross-package schema break — this PR moved populace-calibrate to diagnostics schema 5 but populace-data's publish contract pinned 4 and hard-rejects mismatches (contract.py _check, 'this library publishes version 4'), so the next release build would have gone green and then publish-failed on schema version — the same builder-pass/publisher-reject class #491 just closed, reintroduced across packages.

Pushed the lockstep commit: contract + fixtures to 5, plus test_calibration_diagnostics_schema_lockstep in the builder suite (the one place that imports both packages) so the two same-named constants can never drift silently again. Suites green (data 133, calibrate diagnostics, lockstep). Sol confirm next, then merge.

MaxGhenis and others added 2 commits July 22, 2026 13:13
Sol confirm finding: the census used >= cap, classifying a row exactly
AT the cap as zero-gradient. torch.clamp keeps gradient at the boundary
(verified: d/dx clamp(x, max=cap) at x == cap is 1.0), so an at-cap row
still pulls in the objective. 'Past cap' now means strictly greater,
matching the census's own definition ('zero gradient'), and the test
pins the boundary row as alive.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit 22bd902 into main Jul 22, 2026
4 checks passed
MaxGhenis added a commit that referenced this pull request Aug 13, 2026
…ne solve, validated sidecar (#625)

Downstream-only layer over the ACS local artifact — no national-calibration
diffs anywhere:

- sld_membership: 2024 SLD BEF membership at the ACS target vintage (the
  artifact's baked sldl/sldu are 2020-BAF, donor-only). One seeded operator:
  exact/within-tract lookup for tract-carrying rows, block-overlap draws
  conditional on (PUMA, CD, county) with a recorded fallback ladder for
  ACS-spine rows; both chambers read off one draw. Conservation-checked
  overlap tables built by tools/build_us_sld_membership_ladder_artifact.py
  from pinned sources (NationalSLD*24 BEFs + PL 94-171 + tract-PUMA + cd119).
- sld_local_targets: ledger SLD facts (S0101 age bands, B19001 brackets +
  households, B19013 medians) compiled to per-district problems; medians are
  validation-only by construction; the ACS money-income analog is a declared
  recipe over artifact inputs with declared omissions (SSI, TANF) and
  exclusions.
- sld_local_solver + sld_local_doctrine: independent per-district solves
  reusing populace.calibrate.solve internals; declared cap 10.0 and stretch
  100.0 vs the artifact-weight anchor (#493 stated); structurally knob-free
  doctrine wrappers (#492/#503 ported); per-district past-cap census with
  chamber roll-ups (#494 generalized).
- sld_local_report: achieved-vs-target table, published-median validation,
  statewide coherence (reported, never constrained), and the
  honest-boundaries statement generated into the sidecar.
- tools/build_us_sld_local_layer.py: membership -> targets -> doctrine solve
  -> hashed sidecar chain; reads packaged variable/year and pandas-HDF
  (table + fixed) artifact layouts.
- experiments/sld_local_pilot: first real run on the published #512 rebuild
  (Utah: all 104 districts, median within-10% = 1.00 both chambers, zero
  frozen/pushed-out rows, median-income sanity median gap 3.3%). The binding
  100x ratio and thin-district ESS are recorded for the candidate review.

46 new tests; ruff clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant