Skip to content

Add 2022 SOI sanity-check test, retire superseded variable totals tests (closes most of #501)#508

Merged
donboyd5 merged 1 commit intomasterfrom
soi-sanity-checks
Apr 24, 2026
Merged

Add 2022 SOI sanity-check test, retire superseded variable totals tests (closes most of #501)#508
donboyd5 merged 1 commit intomasterfrom
soi-sanity-checks

Conversation

@donboyd5
Copy link
Copy Markdown
Collaborator

Summary

This is the SOI sanity-check piece described in umbrella #501 (PR 2 in that plan), plus the cleanup deletions that depended on it. It introduces a focused test that anchors five 2022 weighted totals to externally published Statistics of Income (SOI) figures, and retires three pre-existing tests whose roles it covers.

The five checks each pass with comfortable headroom against current TMD data (gaps of 0.1–0.5% against a 1% tolerance).

What's added

tests/test_soi_sanity_2022.py — one test function, five assertions, all with 1% relative tolerance:

Check TMD side SOI side (tmd/storage/input/soi.csv) Observed gap
Number of filers Σ s006 on PUF records count +0.5%
Wages Σ e00200 × s006 on PUF records employment_income −0.3%
AGI Σ c00100 × s006 on PUF records adjusted_gross_income +0.1%
Income tax Σ iitax × s006 (all records) tottax +0.35%
Partnership / S-corp net Σ e26270 × s006 (all records) partnership_and_s_corp_income minus partnership_and_s_corp_losses +0.1%

A note on the income-tax comparator

The comparator for iitax is SOI tottax, not income_tax_after_credits, because both TaxCalc iitax (= c09200 - refund) and SOI tottax include NIIT in scope (and AMT, and other additional taxes except SE and Additional Medicare). Comparing against income_tax_after_credits would show a 2.3% gap that is purely a NIIT-inclusion mismatch, not anything to do with TMD.

A note on PUF-only filtering

Three checks restrict to the PUF subsample (data_source == 1) because the SOI "All filers, full population" target is a filer-population concept. CPS-only records in TMD represent the non-filer population. For the iitax and e26270 checks the mask is unnecessary because CPS records contribute zero to both; the simpler unmask form is used there.

What's removed

  • tests/test_variable_totals.py + tests/taxdata_variable_totals.yaml — superseded entirely. The old test compared against pre-TY2015 taxdata PUF totals with a 45%-or-$30B tolerance — too loose to catch real regressions. The new SOI sanity test replaces that role with current SOI 2022 figures and a 1% tolerance.
  • test_misc.py::test_income_tax (was @pytest.mark.skip) — subsumed by the iitax-vs-tottax check.
  • test_misc.py::test_partnership_s_corp_income (was running) — folded into the new SOI test as the e26270 check, with a tightened 1% tolerance and a sourced SOI target replacing the previous unattributed $975 B / 10% tolerance.

What stays in test_misc.py

  • test_no_negative_weights — narrow invariant check.
  • test_population — Census-anchored population check (distinct external source from SOI; tighter 0.1% tolerance is appropriate).

Test plan

  • make format — clean.
  • make lint — exit 0.
  • pytest tests/test_soi_sanity_2022.py -v — 1 passed in ~13s.
  • make test — 59 passed, 2 skipped (the 2 skipped are the existing markers on test_imputed_variable_distribution and test_tax_revenue, both unchanged by this PR; the previous third skip on test_income_tax is gone because that test was retired).

Related

Adds tests/test_soi_sanity_2022.py — five weighted 2022 totals on the
TMD data, each compared against an externally published Statistics of
Income (SOI) figure with 1% relative tolerance:

  filers (PUF)         vs SOI count                       gap +0.5%
  wages e00200 (PUF)   vs SOI employment_income           gap -0.3%
  AGI c00100 (PUF)     vs SOI adjusted_gross_income       gap +0.1%
  iitax (all rec)      vs SOI tottax                      gap +0.35%
  e26270 (all rec)     vs SOI partnership_and_s_corp_     gap +0.1%
                          income minus _losses

The iitax comparator is SOI tottax (= income tax after credits + NIIT
+ Form 4970 tax) rather than income_tax_after_credits, because both
TaxCalc iitax and SOI tottax include NIIT and AMT in scope. Comparing
against income_tax_after_credits would show a 2.3% gap that is purely
a NIIT-inclusion mismatch, not a TMD problem.

For e26270 (Schedule E line 17, partnership and S-corp net income),
the SOI target is the difference of the two SOI variables since SOI
reports income and losses separately. No PUF-only mask is needed for
iitax or e26270 because CPS records contribute zero to both.

Removes from this PR:

- tests/test_variable_totals.py + tests/taxdata_variable_totals.yaml
  Superseded entirely. The new SOI sanity test replaces its external-
  benchmark role with a 1% tolerance against current SOI 2022 figures
  rather than the old 45%-or-$30B tolerance against pre-TY2015
  taxdata totals.

- test_misc.py::test_income_tax (was @pytest.mark.skip)
  Subsumed by the iitax-vs-tottax check in the new file.

- test_misc.py::test_partnership_s_corp_income (was running)
  Folded into the new SOI test as the e26270 check, with a tightened
  1% tolerance and a sourced SOI target replacing the previous
  unattributed $975 B / 10% tolerance.

Kept in test_misc.py: test_no_negative_weights and test_population.
Both have distinct concepts and external sources from the SOI checks
(min weight invariant; Census population) and stay as standalone
tests.

Test plan:

- make format: clean.
- make lint: exit 0.
- pytest tests/test_soi_sanity_2022.py: 1 passed in ~13s.
- make test: 59 passed, 2 skipped (unchanged set of skip markers
  on test_imputed_variable_distribution and test_tax_revenue;
  test_income_tax skip is gone because the test itself is gone).

Closes part of the work tracked in #501; remaining items in that
umbrella plan will be retired or moved to focused issues separately.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant