Skip to content

Impute capital gains amounts from the published joint distribution - #560

Merged
MaxGhenis merged 2 commits into
mainfrom
uk-cgt-imputation
Aug 13, 2026
Merged

Impute capital gains amounts from the published joint distribution#560
MaxGhenis merged 2 commits into
mainfrom
uk-cgt-imputation

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

Part two of #552, stacked on #558 (the source surface). Review #558 first; this PR's own diff is the last commit.

What the stage does

Redraws existing gainers' capital_gains from HMRC table 3 — the published joint distribution of size of gain by taxable income — so the population can hold the gains above £2m that carry most of the tax. The measured gap this closes: the two top bands hold 54.6% of UK CGT and effectively none of the modelled tax, because the incumbent percentile source stops near £1m.

The stage owns amounts only. Incidence comes from the candidate, household weights pass through untouched (no mass change to record), and whether the published band facts also become calibration targets remains the fence adjudication recorded on the issue — nothing here assumes an answer.

Design

  • Conditioning. HMRC's rows condition on taxable income after reliefs and the Personal Allowance, which the raw-input population does not carry. The proxy sums the persisted components of the model's total_income concept (ITA 2007 s.23) and subtracts a tapered allowance, with the allowance amounts read from the policyengine-uk parameter tree at a stated instant — the uk extra's deferred-import pattern, per its own comment that the base package never imports policyengine-uk at import time. state_pension_reported stands in for social_security_income; reliefs are not deducted. Both divergences are documented where they can move a person one band.
  • Allocation. Rank-preserving within income band: the largest existing gainers absorb the published taxpayer mass, highest gain band first, scaled proportionally where the population holds less gainer mass than HMRC's taxpayers. No person splits across bands, so band mass is matched to one household weight's granularity.
  • Amounts. Within-band distributions matched to each cell's published mean — table 3 publishes a count and an amount per cell, so the mean is data, not assumption. Bounded bands use a truncated exponential (the one-parameter family whose mean can sit anywhere in the band, degenerating to uniform at the midpoint); the open £5m+ band uses a Pareto whose shape the published mean pins. A cell whose count is suppressed falls back to its band-total mean — dividing a published amount by an assumed count can put the implied "mean" outside the band, which is exactly the trap the band totals added in Read HMRC's joint distribution of capital gains by taxable income #558 exist to avoid.
  • Sub-AEA gainers. Table 3 covers only individuals with a CGT liability, so gainers beyond the published taxpayer mass keep their existing amounts capped at the annual exempt amount — not invented into the liability distribution, not deleted. Stated as scope in the module docstring.
  • Determinism. Draws are seeded on the build period; two runs are bit-identical, two periods differ.
  • Reporting. summarize_uk_cgt_imputation compares achieved band totals with the published surface. It is reporting, not a gate: where the population holds less gainer mass than HMRC's taxpayers the achieved totals sit below published by construction, and holding levels to the surface is the calibration question.

What it deliberately does not do

No family_coverage entry, no cgt_source_stages.json, no band-level UK_CGT_TARGET_SPECS. Promoting the stage to required_at_build and declaring targets follow the fence adjudication on #552, on the individuals-only basis (the two existing facts include trusts).

Tests

17 new tests: proxy arithmetic including the taper, mean-matching for both draw families (with the removable singularity at the uniform midpoint covered), determinism, rank preservation, the sub-AEA remainder, suppressed-cell fallback, artifact verification before read, and the parameter-tree read (runs when the uk extra is installed; importorskip otherwise — it is absent from the secrets-free PR gate by design). Full populace-build suite green locally with the extra installed.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator

Heads-up on a planned collision: #612 (carrying UK build datasets as populace Frames) is in motion — the foundations PR #617 is up, and the carrier-swap PR will stack on #610 and re-type UKNationalStage.transform to Frame -> Frame. This PR's cgt_imputation.py transform is typed on UKNationalDataset (and constructs it directly, plus the _dataset() test helper), so whichever lands second pays a small mechanical rebase.

Both orders are pre-agreed fine: if this merges first, the swap absorbs the conversion as one more stage (same pattern as the two production stages); if the swap merges first, I'm happy to push the Frame conversion onto this branch myself so it doesn't sit in your queue. No hold needed either way.

@MaxGhenis

MaxGhenis commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

The swap landed first, so this branch paid the rebase as predicted: 1470a33 merges main, and the adaptation commit carries the stage on the Frame carrier — reads through frame.table, build period from frame metadata, rebuild via uk_national_frame with kind and mass log unchanged (the person-only-replacement idiom from the migrated stages). The adaptation exposed that no test ran the stage factory's own transform (postponed annotations hid a retired type in its signature), so there's now an end-to-end test on the pinned artifact. All 46 CGT tests green; full suite running.

@MaxGhenis

Copy link
Copy Markdown
Contributor Author

An adversarial review pass (cross-model, pre-merge) blocked this with four criticals, all confirmed against the published table. Fixed on the branch; the review trail:

C1 — rounded means outside their band crash the stage. Confirmed: four cells of the 2023-24 table imply a mean below their band floor through rounding — £463m over a rounded 1,000 people reads as £463,000 for the £500k–£1m band (also (£2m–£5m, £50k), (£1m–£2m, £100k), (£250k–£500k, £150k)). The solvers refused those means, so the stage crashed for any population reaching four of the six income bands — while every synthetic test mean sat feasibly in band, and the one real-artifact test used the single income band with no infeasible cell. Implied means are now repaired to just inside the violated boundary (2% of band width), keeping the signal that the cell's mass sits near that edge.

C2 — the 500-person suppression convention overstates mass. Replaced with reconciliation: suppressed cells split the residual between the published income-column total (newly captured from the All row, on #558) and the column's unsuppressed cells. No invented count survives; allocation reconciles to published totals by construction.

C3 — bottom-band draws at or below the AEA. The £0–£10k band straddles the £6,000 AEA, so draws could land people allocated as liable taxpayers below the liability threshold. Its support is now floored at the AEA where that fits inside the band.

C4 — the masking. The real 2023-24 surface is now embedded in the tests as literals (public published statistics), so CI exercises the exact rounding pathology with no artifact present — including a test pinning the repaired £500k-band mean — and the pinned-artifact end-to-end test spans all six income bands instead of the one clean one.

Cleared by the same review: deterministic RNG/ranking, and the Frame weight-kind/mass/linkage invariants.

Rebuilt as one commit on the renamed tree (microcosm.* namespace landed on
main mid-flight, after the Frame carrier swap had already done the same),
carrying the full reviewed history of the stage: the original rank-preserving
amounts-only redraw from HMRC table 3; the Frame carrier adaptation; and two
adversarial review rounds — repair of infeasible rounded means (four 2023-24
cells imply a mean outside their own band), suppressed-count cells taking
the count their published gains imply at the band mean with every income
column rescaled onto its published total, the bottom band's support starting
one pound past the annual exempt amount, and the real published surface
embedded as always-run test literals.

The stage owns amounts, not incidence and not weights. Taxable income is an
arithmetic proxy over persisted total_income components less a tapered
Personal Allowance read from the policyengine-uk parameter tree through the
uk extra's deferred import. Draws are seeded on the build period. Whether
the band facts become calibration targets stays behind the reviewed fence
adjudicated on #552.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The previous round claimed the end-to-end incomes covered all six income
bands; they did not — £45,000 maps to £32,430 after the Personal Allowance,
below the £37,700 bound, so the band went unexercised through the
imputation path in both fixtures. £55,000 reaches it, and the test now
asserts the proxy visits every published band, so the fixture cannot
silently regress the way it silently overclaimed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit faa3dc1 into main Aug 13, 2026
4 checks passed
@MaxGhenis
MaxGhenis deleted the uk-cgt-imputation branch August 13, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants