Add Universal Credit deductions with Fair Repayment Rate cap

[MaxGhenis]

Fixes #1814.

Adds Universal Credit deductions — the money DWP takes off awards to repay advances, government debt (benefit and tax credit overpayments) and third party debts — with the Fair Repayment Rate cap and reform levers. Requested by external model users building deduction reform analysis for charities in the poverty space.

Important

Per-household statistics are the validated layer. Weighted aggregates are not. Counts and costs scale with the model's UC caseload, which runs ~40% below the administrative count (PolicyEngine/policyengine-uk-data#452), so headcounts and exchequer costs of deduction and protected-floor reforms come out low in roughly that proportion. Users replicating published floor estimates should compare per-household gains and distributional shape, not totals. Full limitations: docs/book/validation/uc-deductions.md.

Design: latent demand, statutory cap

Other models assign the observed (post-cap) deduction rates directly, which bakes the cap into the data: raising the cap back to 25% becomes inexpressible. This implementation separates the two:

• uc_latent_deduction_rate assigns deduction demand from the distribution DWP observed in March–May 2025 — the last full months under the 25% cap — in bands of the standard allowance (point masses at institutional rates: 5%, 10%, 15%, 20%, 25%).
• gov.dwp.universal_credit.deductions.cap is a dated parameter (40% to Oct 2019 → 30% → 25% → 15% from 30 April 2025, landing on the repo's April-30 fiscal-year sampling instant) applied at runtime. Latent demand above 25% is last resort deductions (e.g. child maintenance), which the law exempts from the cap.
• uc_deduction_combination assigns which of the three deduction types the household repays, from DWP's published seven-way combination distribution, so type-specific reforms operate household by household.
• Incidence is region-adjusted (South West 0.90× to North East 1.12×) from all 633 constituency rows of DWP's Table 6.

Assignment uses deterministic splitmix64 hashes of benunit_id — reproducible across runs and machines, dispersed like uniform draws, and overridable by datasets or situations (uc_deduction_random_draw, uc_deduction_type_random_draw). Single-household simulations get no deductions unless set explicitly, matching the repo's determinism conventions.

Law vs estimation: statutory parameters (cap, protected floor, abolition switches) live under gov.dwp.universal_credit.deductions; the calibrated distributions live under gov.simulation.uc_deductions — they describe the world, not the law. The assignment formulas are an explicit fallback: the end state imputes these attributes at dataset build (tracked in policyengine-uk-data), at which point the engine consumes them as plain inputs and the fallback retires. Raw-FRS users keep working through the fallback until then.

Reform levers

# Return the cap to 25%:
{"gov.dwp.universal_credit.deductions.cap": {"2026-01-01.2030-12-31": 0.25}}
# Tighten it to 10% (a protected floor at 90% of the standard allowance):
{"gov.dwp.universal_credit.deductions.cap": {"2026-01-01.2030-12-31": 0.10}}
# Abolish government debt deductions only:
{"gov.dwp.universal_credit.deductions.abolish.government": {"2026-01-01.2030-12-31": True}}

A cap of 1 − x is equivalent to a protected minimum floor at x of the standard allowance for capped deduction types. For floor policies that also bind on the benefit cap — the shape of JRF's protected minimum floor, whose post-FRR effect comes mostly through benefit-cap interactions — a dedicated lever limits the combined reductions:

# JRF-style protected minimum floor at 85% of the standard allowance,
# binding on deductions and benefit cap reductions jointly:
{"gov.dwp.universal_credit.deductions.protected_floor": {"2026-01-01.2030-12-31": 0.85}}

The floor limits the combined reductions themselves to (1 − floor) × standard allowance — JRF's design (their worked example caps the reduction at 15% of the standard allowance) — and is zero (inactive, reductions unlimited) under current law.

Validation (enhanced FRS 2023/24 v1.40.3, vs DWP deductions statistics)

statistic	model	DWP published
share of UC households with deductions	47.0–47.1%	46–47%
at-cap share, 25% regime (2024)	13.4%	13–14% (Mar–May 2025)
at-cap share, 15% regime (2025)	26.4%	21% (Jun 2025–Feb 2026)
above-cap share (last resort)	1.8%	2%
mean monthly deduction, 25% regime	£66	£67–68
mean monthly deduction, 15% regime	£50	£51–54
mean annual gain, 25%→15% cap	£421	£420 (gov.uk)
households better off, 25%→15% cap	1.01m	~1.2m (gov.uk)

The pre-FRR at-cap pileup and both regimes' mean amounts reproduce without being separately fitted — the cap parameter generates them from the same latent distribution. Two known gaps, stated rather than tuned away:

• The 15%-regime at-cap share runs ~5pp above observed: the latent model holds demand fixed while real-world deductions attrit under the lower cap (affordability requests, composition drift). This has a direction: reforms that lower the cap further overstate the constrained population and so overstate the gains from later raising it, while reforms that raise the cap toward 25% sit inside the observed window and are on firmer ground.
• Aggregate levels (model: 1.9m deducting households, £1.2bn/yr; admin: 3.3m, ~£2.0bn/yr) scale with the model's UC caseload (4.2m benunits vs ~7.2m admin households) — the known take-up/calibration gap, not a deductions artifact, now tracked as UC caseload runs ~40% below administrative counts, capping aggregate reform estimates policyengine-uk-data#452. Per-household statistics are the validated layer; aggregates inherit whatever the dataset's UC calibration provides. See the callout at the top.
• Only 2024 (25% regime) and 2025 (15% regime) are validated. The latent distribution is observed in 2025 and applied to all model years as a constancy assumption (published deductions statistics begin March 2025), so simulations of years before 2021 apply a 2025-shaped demand distribution under the 40% and 30% caps then in force — defensible forward, uncalibrated backward.

Other limitations

• Rate and type combination are drawn independently (DWP publishes no joint table); type-abolition reforms split a household's rate by national mean amounts per type (£41/£37/£59).
• Deduction spells are annualized (surveyed-status-as-full-year convention); a synthetic-panel treatment could refine this. Monthly means validate, so cross-sectional statistics are sound, but annual per-household gains are upper bounds for spell-limited types such as advance recovery.
• The last resort band is top-coded at 25% in DWP tables; its upper edge (30%) is a modeling assumption.
• Regional factors derive from rounded constituency proportions (method: Σ deducting households / Σ implied UC households by region). The rate distribution itself is national — regional composition plausibly differs (more advances in high-incidence regions) and the module assumes it does not.
• The protected floor binds on the above-cap excess (last resort and child maintenance deductions) that current law exempts from the deductions cap. JRF's briefing does not say whether their floor exempts those categories — its worked example involves only cappable deductions and the benefit cap — so this is a modeling choice, documented on the parameter and pinned by a test rather than left implicit. Open for follow-up if JRF's design turns out to exempt them.

Tests

• 7 policy YAML tests: calculator defaults (including high-incidence regions), 25% vs 15% cap on identical latent demand, the last resort excess, and the one-penny safeguard.
• 13 unit tests: hash determinism/uniformity/stream independence, calculator defaults, cap restoration reform, type abolition scaling, zero-mean reform guard, the protected floor (binding through the benefit cap; binding on the above-cap excess; inactive when reductions stay within it), and the all-shares-zero branch of uc_deduction_combination still yielding an EnumArray.
• 2 microsimulation regression tests pinning the per-household statistics above with drift-tolerant bands.
• Full policy suite passes (1,116 tests); 135 unit tests pass locally. Microsimulation tests need the gated dataset and run in CI.

Documentation

New page docs/book/validation/uc-deductions.md: the validation table, every limitation above with its practical consequence for reform estimates, and the reform levers.

🤖 Generated with Claude Code

[MaxGhenis]

The Test job failure is environmental, not from this branch: every error in both runs is the same Hugging Face 401 Unauthorized downloading the private dataset (policyengine-uk-data-private/enhanced_frs_2023_24.h5@1.40.3) — there are no assertion failures in either log (first run, rerun). The full suite passes locally against the same pinned dataset (1,115 policy tests + the new unit, YAML and microsimulation tests). Main's last push CI was green on 27 July, so the HUGGING_FACE_TOKEN Actions secret broke somewhere in the last week — filing an infra issue; any dataset-backed PR will be red until it's fixed.

[MaxGhenis]

Adversarial review round (GPT-5.6 sol, PolicyEngine review-program workflow) returned seven confirmed findings, all fixed in 9164354:

1. Protected floor semantics — now caps combined reductions at (1 − floor) × standard allowance per JRF's worked example (reduction limited to 15% of SA), instead of flooring the payment against total entitlement; arithmetically identical to the pre-floor formula when zero.
2. Last resort exemption — latent demand above 25% now splits into a cappable portion plus a last-resort excess that sits on top of the cap, instead of exempting the household's entire deduction; post-FRR above-cap share moves from 2.9% to 1.8% (DWP: 2%).
3. One-penny safeguard — deductions leave 1p per assessment period payable (SI 2013/380 Schedule 6).
4. Regional factors rebuilt — Table 6 labels each region only on its first row; the earlier parse kept one constituency per region. Re-derived from all 633 rows: North West 0.95→1.09, East of England 1.06→0.93, and the rest.
5. Calculator defaults — draws default to 1.0, so high-incidence regions (North East ×1.12) no longer get deductions in single-household simulations.
6. Distribution shares — AT_15/AT_25 corrected to the true three-month averages (.047/.133); observation-period and modeling-assumption provenance documented on every estimation parameter.
7. Hardening — zero-mean division guard; splitmix draws use the top 53 bits (strictly below 1); statutory references added (Schedule 6, Budget 2018/2020 announcements, DWP guide page anchor).

Documented rather than changed: the 1e6-weight microdata heuristic and float32 draw storage (noted limitations), and the 15%-regime at-cap gap (model 26.4% vs observed 21% — the latent-demand model holds demand fixed where reality attrits under the lower cap; stated in the body).

Post-fix validation improved on five of seven headline statistics (table in the PR body updated).

[MaxGhenis]

Verification round (sol, round 2) confirmed all seven round-1 fixes mechanically — including independently reproducing the regional derivation from the official ODS to 3e-16 and probe-testing the floor/split/penny across edge cases — and surfaced a second tier, fixed in the latest commit: the one-penny amount and last-resort boundary are now parameters (minimum_payable, calibration_cap) rather than formula constants; cap provenance corrected (Schedule 6 para 4(1) is a 40% statutory ceiling with operational caps set administratively — the FRR needed no legislative change; 25% was Budget 2020's October 2021 date brought forward at Budget 2021); float32 storage endpoint clamped; docstrings and reference page anchors updated. Local suite: 1,116 policy + 11 unit + 7 YAML + microsim/convention tests green.

[vahid-ahmadi]

Reviewed for methodology, data provenance, and validity. The core design — latent demand calibrated to the last 25%-cap window, statutory cap applied at runtime, last-resort excess exempt — is sound, and the validation table (pre-FRR pileup and both regimes' means reproducing from one latent distribution without separate fitting) is convincing at the per-household level. Specific points:

Correctness/semantics

1. Protected floor folds the above-cap excess in. universal_credit limits benefit_cap_reduction + uc_deductions jointly, and uc_deductions includes the last-resort/child-maintenance excess. Under current law those categories can exceed the cap; whether JRF's floor is intended to bind on them too isn't obvious from their briefing (their worked example only involves cappable deductions and the benefit cap). Worth a one-line note on protected_floor.yaml that the modeled floor binds on all deductions including last-resort — and, if JRF's design exempts them, a follow-up.
2. Rate/incidence share one draw. uc_latent_deduction_rate reuses uc_deduction_random_draw rescaled by regional incidence — conditional on assignment the quantile is uniform, so this is correct, but it hard-couples region to the rate quantile: in high-incidence regions the marginal households assigned by the regional uplift all land in the top of the rate distribution... actually they land uniformly since draw/incidence is uniform on [0,1) — fine. But note the rate distribution itself is national; regions plausibly differ in composition (more advances in high-incidence regions). Documented constancy assumption would help.
3. uc_deduction_combination zero-total branch returns np.full(..., dtype=object) while the normal branch returns a select over Enum values — check this degrades gracefully to an EnumArray; a test reforming all seven shares to zero would cover it.
4. Historic years inherit the 2025 latent distribution (values from 2015-04-01) while the cap parameter is 40%/30% back then — so 2019–2024 simulations produce deduction levels from a 2025-shaped demand distribution truncated at higher caps. Defensible forward, not calibrated backward; the descriptions say "constancy assumption" but a sentence in the PR body on which years are considered validated (2024–2025 only) would prevent misquoting.

Validity for the target use cases

5. Aggregates are the binding limitation for JRF-style claims. ~1.9m deducting households vs 3.3m admin means poverty counts and £ costs of a floor reform will run ~40% low regardless of per-household validation. Since the requesting users (WPI, charities) will want exactly those aggregates, I'd promote this from a limitations bullet to a prominent caveat in the changelog/docs, with a pointer to the caseload calibration issue in policyengine-uk-data.
6. 15%-regime at-cap gap (26.4% vs 21%): holding latent demand fixed under the lower cap is the right first-order choice, and the direction of bias is stated. Implication worth adding: reforms that lower the cap further will overstate the number of constrained households and thus overstate gains; reforms raising the cap are on firmer ground since the latent distribution was observed under 25%.
7. Annualization inflates annual amounts for short spells (advance recovery ≤24 months). Monthly means validate, so cross-sectional statistics are fine; annual per-household gains are upper bounds for spell-limited types.

Smaller items

• retained(kept, total) uses Python max(total, 1e-9) on what are parameter scalars — fine as written since m.* are scalars, just noting it would break if these ever became arrays.
• The 1e6-weight heuristic distinguishing microdata from calculators is pragmatic but silent: a dataset with unweighted benunits would get calculator behavior. The documented escape hatch (setting draws explicitly) covers it; the planned move of imputation to dataset build retires the whole issue — good end-state.
• Tests are genuinely strong: the cap-restoration reform test and the floor binding/non-binding pair cover the reform levers practitioners will actually pull.

Overall: methodology is publishable with the aggregate-caseload caveat made prominent; per-household statistics are the quotable layer, matching how the validation is framed.

[MaxGhenis]

Thanks for the review — all seven points are addressed in #1821: floor-scope note on protected_floor.yaml (binds on last resort + child maintenance; flagged for follow-up if JRF's design exempts them), national-distribution and validated-years (2024–25) documentation, the direction-of-bias note for cap reforms, the annualization upper-bound note, the aggregate caveat promoted to the changelog with pointers to policyengine-uk-data#452 and #450, and a graceful-degradation test for zeroed combination shares. The scalar-max and weight-heuristic observations are noted in place; both retire with the dataset-build migration (#450).