diff --git a/.github/workflows/code_changes.yaml b/.github/workflows/code_changes.yaml index 6a0008f2e..6cef7d63e 100644 --- a/.github/workflows/code_changes.yaml +++ b/.github/workflows/code_changes.yaml @@ -54,6 +54,13 @@ jobs: # Only export token if non-empty (defensive against missing secrets) if [ -n "$HF_TOKEN_VALUE" ]; then export HUGGING_FACE_TOKEN="$HF_TOKEN_VALUE" + # policyengine-core only passes a token to hf_hub_download when + # the repo reports private=True. policyengine-uk-data-private + # has been public + gated since 31 July 2026, so core sends the + # request anonymously and the gate returns 401. HF_TOKEN is + # huggingface_hub's implicit-token variable, used when the + # explicit token is None, which restores authentication. + export HF_TOKEN="$HF_TOKEN_VALUE" fi make test env: diff --git a/.github/workflows/pr_code_changes.yaml b/.github/workflows/pr_code_changes.yaml index 17c973b5e..769ff3501 100644 --- a/.github/workflows/pr_code_changes.yaml +++ b/.github/workflows/pr_code_changes.yaml @@ -73,6 +73,13 @@ jobs: # Only export token if non-empty (avoids 'Bearer ' error for Dependabot PRs) if [ -n "$HF_TOKEN_VALUE" ]; then export HUGGING_FACE_TOKEN="$HF_TOKEN_VALUE" + # policyengine-core only passes a token to hf_hub_download when + # the repo reports private=True. policyengine-uk-data-private + # has been public + gated since 31 July 2026, so core sends the + # request anonymously and the gate returns 401. HF_TOKEN is + # huggingface_hub's implicit-token variable, used when the + # explicit token is None, which restores authentication. + export HF_TOKEN="$HF_TOKEN_VALUE" fi make test env: diff --git a/changelog.d/uc-deductions.added.md b/changelog.d/uc-deductions.added.md new file mode 100644 index 000000000..d4359928b --- /dev/null +++ b/changelog.d/uc-deductions.added.md @@ -0,0 +1 @@ +Universal Credit deductions: latent deduction demand assigned from DWP deductions statistics (incidence by region, rate distribution, type combinations), the deductions cap including the 2025 Fair Repayment Rate, last resort deductions exempt from the cap, reform switches to abolish advance, third party or government debt deductions, and a protected minimum floor lever limiting combined deductions and benefit cap reductions (JRF-style floor reforms). Per-household statistics are validated against the DWP deductions statistics; weighted aggregates (deducting households, total deducted, the cost of floor reforms) run low in proportion to the model's UC caseload shortfall (policyengine-uk-data#452) and should not be quoted without that caveat - see the deductions validation page in the documentation for the limitations that bound reform estimates. diff --git a/docs/book/validation/uc-deductions.md b/docs/book/validation/uc-deductions.md new file mode 100644 index 000000000..5d7d4fc4a --- /dev/null +++ b/docs/book/validation/uc-deductions.md @@ -0,0 +1,84 @@ +# Universal Credit deductions + +```{important} +Per-household deduction statistics are validated against the DWP deductions statistics. **Weighted aggregates are not.** They scale with the model's Universal Credit caseload, which falls short of the administrative count, so headcounts and costs of deduction reforms — including protected-floor reforms — run materially low. Quote the per-household layer; treat aggregates as lower bounds until the caseload gap closes. +``` + +PolicyEngine UK models the money DWP takes off Universal Credit awards to repay debts: advance repayments, government debt (benefit and tax credit overpayments) and third party deductions such as utility arrears. + +The implementation separates **latent demand** from the **statutory cap**. `uc_latent_deduction_rate` assigns deduction demand from the distribution DWP observed in March to May 2025 — the last full months under the 25% cap — and `gov.dwp.universal_credit.deductions.cap` applies the operational cap (40% to October 2019, 30%, 25%, then 15% under the Fair Repayment Rate from 30 April 2025) at runtime. Assigning observed post-cap rates directly would bake the cap into the data and make cap-raising reforms inexpressible. + +## Validation against DWP deductions statistics + +Enhanced FRS 2023/24 v1.40.3, against [DWP's Universal Credit deductions statistics](https://www.gov.uk/government/statistics/universal-credit-quarterly-statistics-29-april-2013-to-12-february-2026/universal-credit-deductions-statistics-march-2025-to-february-2026). + +| Statistic | Model | DWP published | +|---|---|---| +| Share of UC households with deductions | 47.0–47.1% | 46–47% | +| At-cap share, 25% regime (2024) | 13.4% | 13–14% (Mar–May 2025) | +| At-cap share, 15% regime (2025) | 26.4% | 21% (Jun 2025–Feb 2026) | +| Above-cap share (last resort) | 1.8% | 2% | +| Mean monthly deduction, 25% regime | £66 | £67–68 | +| Mean monthly deduction, 15% regime | £50 | £51–54 | +| Mean annual gain, 25%→15% cap | £421 | £420 (gov.uk) | +| Households better off, 25%→15% cap | 1.01m | ~1.2m (gov.uk) | + +The pre-Fair-Repayment-Rate pileup at the cap and both regimes' mean amounts reproduce without being separately fitted: the cap parameter generates them from the same latent distribution. + +## Limitations + +These bound what the module can support. They are stated rather than tuned away. + +### Aggregates scale with the UC caseload + +The model has 4.2m UC benefit units against roughly 7.2m administrative households, and correspondingly 1.9m deducting households and £1.2bn/year deducted against 3.3m and roughly £2.0bn/year in the administrative data. This is the model's UC take-up and calibration gap, not a deductions artefact — per-household statistics validate precisely because the gap divides out. It is tracked in [policyengine-uk-data#452](https://github.com/PolicyEngine/policyengine-uk-data/issues/452). + +The practical consequence: **poverty headcounts and exchequer costs of a protected-floor or cap reform run low roughly in proportion to the caseload shortfall.** Analyses replicating published floor estimates (such as JRF's) should compare per-household gains and distributional shape, not totals. + +### The at-cap share under the 15% regime runs high + +The model puts 26.4% of UC households at the 15% cap against 21% observed. The latent-demand model holds demand fixed while real deductions attrit under a lower cap through affordability requests and composition drift. + +This has a direction. Reforms that **lower** the cap further overstate the number of constrained households and so overstate the gains from later raising it. Reforms that **raise** the cap toward 25% sit inside the observed window and are on firmer ground. + +### Only 2024 and 2025 are validated + +The latent distribution is observed in 2025 and applied to all model years as a constancy assumption; published deductions statistics begin in March 2025. Simulations of years before 2021 apply a 2025-shaped demand distribution under the 40% and 30% caps then in force, which no published statistic can check. Treat pre-2024 deduction results as uncalibrated. + +### The protected floor binds on last resort deductions + +The `protected_floor` lever limits combined deductions and benefit cap reductions to (1 − floor) × the standard allowance. It binds on the whole of `uc_deductions`, including the above-cap excess that current law exempts from the deductions cap (last resort and child maintenance deductions). JRF's briefing does not say whether their floor exempts those categories — its worked example involves only cappable deductions and the benefit cap — so this is a modeling choice. + +### Other assumptions + +- Rate and type combination are drawn independently: DWP publishes no joint table. Type-abolition reforms split a household's rate by national mean amounts per type (£41 advance / £37 third party / £59 government). +- Deduction spells are annualised under the surveyed-status-as-full-year convention. Monthly means validate, so cross-sectional statistics are sound, but annual per-household gains are upper bounds for spell-limited types such as advance recovery (at most 24 months). +- The last resort band is top-coded at 25% in DWP tables; its 30% upper edge is a modeling assumption. +- Regional incidence factors derive from rounded constituency proportions across all 633 rows of DWP's Table 6. +- The rate distribution is national. Regional composition plausibly differs (more advances in high-incidence regions); the module assumes it does not. + +## Reform levers + +```python +# Return the cap to 25%: +{"gov.dwp.universal_credit.deductions.cap": {"2026-01-01.2030-12-31": 0.25}} + +# Abolish government debt deductions only: +{ + "gov.dwp.universal_credit.deductions.abolish.government": { + "2026-01-01.2030-12-31": True + } +} + +# JRF-style protected minimum floor at 85% of the standard allowance, +# binding on deductions and benefit cap reductions jointly: +{"gov.dwp.universal_credit.deductions.protected_floor": {"2026-01-01.2030-12-31": 0.85}} +``` + +A cap of 1 − *x* is equivalent to a protected minimum floor at *x* of the standard allowance for capped deduction types. The dedicated `protected_floor` lever additionally binds on benefit cap reductions, which is where most of JRF's post-Fair-Repayment-Rate effect comes from. + +## Where the assignment lives + +Statutory parameters — the cap, the protected floor, the minimum payable penny, the abolition switches — live under `gov.dwp.universal_credit.deductions`. The calibrated distributions live under `gov.simulation.uc_deductions`: they describe the world, not the law. + +The assignment formulas are an explicit fallback. The end state imputes `uc_latent_deduction_rate` and `uc_deduction_combination` at dataset build, at which point the model consumes them as plain inputs and the fallback retires; raw-FRS users keep working through the fallback until then. Assignment uses deterministic splitmix64 hashes of `benunit_id`, reproducible across runs and machines, and overridable by datasets or situations through `uc_deduction_random_draw` and `uc_deduction_type_random_draw`. Single-household simulations get no deductions unless set explicitly. diff --git a/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/abolish/advance.yaml b/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/abolish/advance.yaml new file mode 100644 index 000000000..eee4badd2 --- /dev/null +++ b/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/abolish/advance.yaml @@ -0,0 +1,9 @@ +description: Remove advance repayment deductions from Universal Credit. Reform switch - false under current law, which recovers advances by deduction under Schedule 6 of SI 2013/380 and DWP deductions guidance. +values: + 2015-04-01: false +metadata: + label: Abolish UC advance repayment deductions + unit: bool + reference: + - title: DWP guidance, Deductions V17.0 (deduction types and priority order) + href: https://data.parliament.uk/DepositedPapers/Files/DEP2024-0673/054_Deductions_V17.0.pdf#page=3 diff --git a/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/abolish/government.yaml b/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/abolish/government.yaml new file mode 100644 index 000000000..405f33b08 --- /dev/null +++ b/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/abolish/government.yaml @@ -0,0 +1,9 @@ +description: Remove government debt deductions (benefit and tax credit overpayment recovery) from Universal Credit. Reform switch - false under current law, which recovers government debt by deduction under Schedule 6 of SI 2013/380 and DWP deductions guidance. +values: + 2015-04-01: false +metadata: + label: Abolish UC government debt deductions + unit: bool + reference: + - title: DWP guidance, Deductions V17.0 (deduction types and priority order) + href: https://data.parliament.uk/DepositedPapers/Files/DEP2024-0673/054_Deductions_V17.0.pdf#page=2 diff --git a/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/abolish/third_party.yaml b/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/abolish/third_party.yaml new file mode 100644 index 000000000..cf3a80dd7 --- /dev/null +++ b/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/abolish/third_party.yaml @@ -0,0 +1,9 @@ +description: Remove third party deductions (e.g. utility and housing arrears paid direct to creditors) from Universal Credit. Reform switch - false under current law, which makes third party deductions under Schedule 6 of SI 2013/380 and DWP deductions guidance. +values: + 2015-04-01: false +metadata: + label: Abolish UC third party deductions + unit: bool + reference: + - title: DWP guidance, Deductions V17.0 (deduction types and priority order) + href: https://data.parliament.uk/DepositedPapers/Files/DEP2024-0673/054_Deductions_V17.0.pdf#page=4 diff --git a/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/cap.yaml b/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/cap.yaml new file mode 100644 index 000000000..9a62d98a8 --- /dev/null +++ b/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/cap.yaml @@ -0,0 +1,27 @@ +description: Maximum total deductions from a Universal Credit award, as a fraction of the standard allowance. Schedule 6 paragraph 4(1) of SI 2013/380 sets a statutory ceiling of 40%; within it, DWP sets the operational cap administratively - 30% from October 2019 (Budget 2018), 25% from April 2021 (announced for October 2021 at Budget 2020, brought forward at Budget 2021), and 15% (the Fair Repayment Rate) from 30 April 2025, which required no legislative change. Last resort deductions (housing and fuel arrears hardship deductions) and child maintenance deductions - separate categories in DWP guidance - may each exceed the cap (Explanatory Memorandum to SI 2025/387, paragraphs 5.10 to 5.15). +values: + 2015-04-01: 0.4 + 2019-10-01: 0.3 + 2021-04-01: 0.25 + 2025-04-30: 0.15 +metadata: + label: UC deductions cap + unit: /1 + period: year + reference: + - title: SI 2013/380, Schedule 6, paragraph 4(1) (40% statutory ceiling) + href: https://www.legislation.gov.uk/uksi/2013/380/schedule/6/paragraph/4/1 + - title: Explanatory Memorandum to SI 2025/387, paragraph 5.2 (operational cap set within the ceiling; no legislative change for the Fair Repayment Rate) + href: https://www.legislation.gov.uk/uksi/2025/387/pdfs/uksiem_20250387_en_001.pdf#page=2 + - title: Explanatory Memorandum to SI 2025/387, paragraphs 5.10 to 5.15 (child maintenance deductions may exceed the 15% cap) + href: https://www.legislation.gov.uk/uksi/2025/387/pdfs/uksiem_20250387_en_001.pdf#page=3 + - title: DWP guidance, Deductions V17.0 (maximum rates of deductions) + href: https://data.parliament.uk/DepositedPapers/Files/DEP2024-0673/054_Deductions_V17.0.pdf#page=3 + - title: HM Treasury, Budget 2018, para 5.36 (cap to 30% from October 2019) + href: https://assets.publishing.service.gov.uk/media/5bd718b8ed915d78a0dc107d/Budget_2018_red_web.pdf#page=80 + - title: HM Treasury, Budget 2020, para 2.9 (cap to 25%, announced for October 2021) + href: https://assets.publishing.service.gov.uk/media/5e68d0dce90e070ac74f08ea/Budget_2020_Web_Accessible_Complete.pdf#page=73 + - title: HM Treasury, Budget 2021 (25% cap brought forward to April 2021) + href: https://www.gov.uk/government/publications/budget-2021-documents/budget-2021-html + - title: Universal Credit change brings £420 boost to over a million households (Fair Repayment Rate, 30 April 2025) + href: https://www.gov.uk/government/news/universal-credit-change-brings-420-boost-to-over-a-million-households diff --git a/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/minimum_payable.yaml b/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/minimum_payable.yaml new file mode 100644 index 000000000..075b067fe --- /dev/null +++ b/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/minimum_payable.yaml @@ -0,0 +1,10 @@ +description: Minimum Universal Credit payable after deductions, per assessment period. Schedule 6 of SI 2013/380 provides that deductions must not reduce the amount payable to the claimant below one penny. +values: + 2015-04-01: 0.01 +metadata: + label: UC minimum payable after deductions + unit: currency-GBP + period: month + reference: + - title: SI 2013/380, Schedule 6 (deductions from benefit and direct payment to third parties) + href: https://www.legislation.gov.uk/uksi/2013/380/schedule/6 diff --git a/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/protected_floor.yaml b/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/protected_floor.yaml new file mode 100644 index 000000000..50fbf3aca --- /dev/null +++ b/policyengine_uk/parameters/gov/dwp/universal_credit/deductions/protected_floor.yaml @@ -0,0 +1,10 @@ +description: Protected minimum floor in Universal Credit, as a fraction of the standard allowance. When above zero, combined deductions and benefit cap reductions are limited to (1 - floor) times the standard allowance, following JRF's proposal design (their worked example limits the reduction itself to 15% of the standard allowance). Zero - the value under current law, which has no such floor - leaves reductions unlimited. Setting 0.85 models JRF's proposal; this reform-lever parameter encodes no current-law value other than zero. The modeled floor binds on all deductions, including the above-cap excess that current law exempts from the deductions cap (last resort and child maintenance deductions) - JRF's briefing does not state whether their floor exempts those categories, and its worked example involves only cappable deductions and the benefit cap, so treating the floor as binding on everything is a modeling choice rather than a stated feature of the proposal. +values: + 2015-04-01: 0 +metadata: + label: UC protected minimum floor + unit: /1 + period: year + reference: + - title: JRF, A protected minimum floor in Universal Credit (October 2025 briefing; proposal design source, not current law) + href: https://www.jrf.org.uk/social-security/protected-minimum-floor-in-universal-credit-new-policy-targeting-hardship diff --git a/policyengine_uk/parameters/gov/simulation/uc_deductions/calibration_cap.yaml b/policyengine_uk/parameters/gov/simulation/uc_deductions/calibration_cap.yaml new file mode 100644 index 000000000..390f7a701 --- /dev/null +++ b/policyengine_uk/parameters/gov/simulation/uc_deductions/calibration_cap.yaml @@ -0,0 +1,10 @@ +description: The deductions cap in force during the calibration window (March to May 2025, under the 25% cap). Latent demand above this boundary is the above-cap excess (last resort and child maintenance categories), which sits on top of the operational cap; demand at exactly this level is the old-cap pileup, treated as cappable. Changes only if the model is recalibrated to a window under a different cap. +values: + 2015-04-01: 0.25 +metadata: + label: UC deductions calibration-window cap + unit: /1 + period: year + reference: + - title: DWP, Universal Credit deductions statistics March 2025 to February 2026, supplementary data tables, Table 2 + href: https://assets.publishing.service.gov.uk/media/69fb3ea22a6137e93226b7ce/universal-credit-deductions-march-2025-to-february-2026.ods diff --git a/policyengine_uk/parameters/gov/simulation/uc_deductions/latent_rate_distribution.yaml b/policyengine_uk/parameters/gov/simulation/uc_deductions/latent_rate_distribution.yaml new file mode 100644 index 000000000..02542f65d --- /dev/null +++ b/policyengine_uk/parameters/gov/simulation/uc_deductions/latent_rate_distribution.yaml @@ -0,0 +1,83 @@ +description: Distribution of latent Universal Credit deduction demand, as a share of all UC claims in each deduction-rate band (rates as a fraction of the standard allowance). Simple average of the observed March to May 2025 distribution (Table 2), the last full months under the 25% cap, rounded to three decimals; the cap parameter reproduces observed post-cap distributions when applied. Bands sum to the share of UC claims with any deduction (0.467). Observed 2025 and applied to all model years as a constancy assumption (published deductions statistics begin March 2025). Treating the at-25% pileup as latent demand of at least 25%, and observations above 25% as above-cap deductions (last resort and child maintenance categories) with a 30% modeled upper edge, are modeling assumptions, not source facts. Validated for 2024 (25% cap regime) and 2025 (15% cap regime) only - earlier years apply this 2025-shaped demand distribution under the higher caps then in force (40% to October 2019, 30% to April 2021), which the published statistics cannot check. Demand is held fixed when the cap changes, so reforms that lower the cap below 15% overstate the number of constrained households and hence the gains from raising it back, while reforms that raise the cap toward 25% sit inside the observed window and are on firmer ground. +UNDER_5: + description: Share of UC households with deductions between 0% and 5% of the standard allowance. + values: + 2015-04-01: 0.047 + metadata: + label: UC deduction share, under 5% of standard allowance + unit: /1 +AT_5: + description: Share of UC households with deductions at exactly 5% of the standard allowance. + values: + 2015-04-01: 0.030 + metadata: + label: UC deduction share, at 5% of standard allowance + unit: /1 +FIVE_TO_10: + description: Share of UC households with deductions between 5% and 10% of the standard allowance. + values: + 2015-04-01: 0.070 + metadata: + label: UC deduction share, 5% to 10% of standard allowance + unit: /1 +AT_10: + description: Share of UC households with deductions at exactly 10% of the standard allowance. + values: + 2015-04-01: 0.010 + metadata: + label: UC deduction share, at 10% of standard allowance + unit: /1 +TEN_TO_15: + description: Share of UC households with deductions between 10% and 15% of the standard allowance. + values: + 2015-04-01: 0.040 + metadata: + label: UC deduction share, 10% to 15% of standard allowance + unit: /1 +AT_15: + description: Share of UC households with deductions at exactly 15% of the standard allowance. + values: + 2015-04-01: 0.047 + metadata: + label: UC deduction share, at 15% of standard allowance + unit: /1 +FIFTEEN_TO_20: + description: Share of UC households with deductions between 15% and 20% of the standard allowance. + values: + 2015-04-01: 0.030 + metadata: + label: UC deduction share, 15% to 20% of standard allowance + unit: /1 +AT_20: + description: Share of UC households with deductions at exactly 20% of the standard allowance. + values: + 2015-04-01: 0.010 + metadata: + label: UC deduction share, at 20% of standard allowance + unit: /1 +TWENTY_TO_25: + description: Share of UC households with deductions between 20% and 25% of the standard allowance. + values: + 2015-04-01: 0.030 + metadata: + label: UC deduction share, 20% to 25% of standard allowance + unit: /1 +AT_25: + description: Share of UC households with latent deduction demand at or above 25% of the standard allowance (observed at the 25% cap before the Fair Repayment Rate). + values: + 2015-04-01: 0.133 + metadata: + label: UC deduction share, at or above 25% of standard allowance + unit: /1 +OVER_25: + description: Share of UC households with deductions above 25% of the standard allowance (categories permitted to exceed the cap, such as last resort and child maintenance deductions). + values: + 2015-04-01: 0.020 + metadata: + label: UC deduction share, above 25% of standard allowance (above-cap categories) + unit: /1 +metadata: + label: UC latent deduction rate distribution + reference: + - title: DWP, Universal Credit deductions statistics March 2025 to February 2026, supplementary data tables, Table 2 + href: https://assets.publishing.service.gov.uk/media/69fb3ea22a6137e93226b7ce/universal-credit-deductions-march-2025-to-february-2026.ods diff --git a/policyengine_uk/parameters/gov/simulation/uc_deductions/mean_monthly_amount_by_type.yaml b/policyengine_uk/parameters/gov/simulation/uc_deductions/mean_monthly_amount_by_type.yaml new file mode 100644 index 000000000..bba8d4086 --- /dev/null +++ b/policyengine_uk/parameters/gov/simulation/uc_deductions/mean_monthly_amount_by_type.yaml @@ -0,0 +1,30 @@ +description: Mean monthly deduction amount by deduction type among Universal Credit households with that type, from the March to May 2025 average (Table 4), applied to all model years as a constancy assumption. These are marginal means; using them to split a household's total deduction across its types in abolition reforms is a modeling approximation, since DWP publish no type-by-amount joint table. +ADVANCE: + description: Mean monthly advance repayment deduction. + values: + 2015-04-01: 41 + metadata: + label: UC mean advance deduction + unit: currency-GBP + period: month +THIRD_PARTY: + description: Mean monthly third party deduction. + values: + 2015-04-01: 37 + metadata: + label: UC mean third party deduction + unit: currency-GBP + period: month +GOVERNMENT: + description: Mean monthly government debt deduction. + values: + 2015-04-01: 59 + metadata: + label: UC mean government deduction + unit: currency-GBP + period: month +metadata: + label: UC mean deduction amount by type + reference: + - title: DWP, Universal Credit deductions statistics March 2025 to February 2026, supplementary data tables, Table 4 + href: https://assets.publishing.service.gov.uk/media/69fb3ea22a6137e93226b7ce/universal-credit-deductions-march-2025-to-february-2026.ods diff --git a/policyengine_uk/parameters/gov/simulation/uc_deductions/region_incidence_factor.yaml b/policyengine_uk/parameters/gov/simulation/uc_deductions/region_incidence_factor.yaml new file mode 100644 index 000000000..4ba6b42eb --- /dev/null +++ b/policyengine_uk/parameters/gov/simulation/uc_deductions/region_incidence_factor.yaml @@ -0,0 +1,84 @@ +description: Regional deduction incidence relative to the national rate. Derived from all 633 constituency rows of Table 6 of the DWP deductions supplementary tables (February 2026), with region labels forward-filled from the table's first-row-per-region layout; each region's share is the household-weighted mean (deducting households divided by implied UC households, from counts and rounded proportions), divided by the national share computed the same way (0.464). Observed February 2026 and applied to all model years as a constancy assumption. Unknown region and Northern Ireland (administered separately, absent from the GB statistics) use the national rate by assumption. +UNKNOWN: + values: + 2015-04-01: 1 + metadata: + label: UC deduction incidence factor, unknown region + unit: /1 +NORTH_EAST: + values: + 2015-04-01: 1.12 + metadata: + label: UC deduction incidence factor, North East + unit: /1 +NORTH_WEST: + values: + 2015-04-01: 1.09 + metadata: + label: UC deduction incidence factor, North West + unit: /1 +YORKSHIRE: + values: + 2015-04-01: 1.05 + metadata: + label: UC deduction incidence factor, Yorkshire and the Humber + unit: /1 +EAST_MIDLANDS: + values: + 2015-04-01: 0.97 + metadata: + label: UC deduction incidence factor, East Midlands + unit: /1 +WEST_MIDLANDS: + values: + 2015-04-01: 1 + metadata: + label: UC deduction incidence factor, West Midlands + unit: /1 +EAST_OF_ENGLAND: + values: + 2015-04-01: 0.93 + metadata: + label: UC deduction incidence factor, East of England + unit: /1 +LONDON: + values: + 2015-04-01: 0.94 + metadata: + label: UC deduction incidence factor, London + unit: /1 +SOUTH_EAST: + values: + 2015-04-01: 0.93 + metadata: + label: UC deduction incidence factor, South East + unit: /1 +SOUTH_WEST: + values: + 2015-04-01: 0.9 + metadata: + label: UC deduction incidence factor, South West + unit: /1 +WALES: + values: + 2015-04-01: 1.05 + metadata: + label: UC deduction incidence factor, Wales + unit: /1 +SCOTLAND: + values: + 2015-04-01: 1.09 + metadata: + label: UC deduction incidence factor, Scotland + unit: /1 +NORTHERN_IRELAND: + values: + 2015-04-01: 1 + metadata: + label: UC deduction incidence factor, Northern Ireland + unit: /1 +metadata: + label: UC deduction regional incidence factors + reference: + - title: DWP, Universal Credit deductions statistics March 2025 to February 2026, supplementary data tables, Table 6 + href: https://assets.publishing.service.gov.uk/media/69fb3ea22a6137e93226b7ce/universal-credit-deductions-march-2025-to-february-2026.ods diff --git a/policyengine_uk/parameters/gov/simulation/uc_deductions/type_combination.yaml b/policyengine_uk/parameters/gov/simulation/uc_deductions/type_combination.yaml new file mode 100644 index 000000000..934c9c25d --- /dev/null +++ b/policyengine_uk/parameters/gov/simulation/uc_deductions/type_combination.yaml @@ -0,0 +1,55 @@ +description: Distribution of deduction type combinations among Universal Credit households with any deduction, from the March to May 2025 average of Table 5 (advances, third party deductions, and government deductions), applied to all model years as a constancy assumption. Encoded shares sum to 1.004 because the published whole-point rows can sum above 100%; formulas normalize. +ADVANCE_ONLY: + description: Share of UC households with deductions repaying only advances. + values: + 2015-04-01: 0.327 + metadata: + label: UC deduction combination share, advances only + unit: /1 +THIRD_PARTY_ONLY: + description: Share of UC households with deductions with only third party deductions. + values: + 2015-04-01: 0.060 + metadata: + label: UC deduction combination share, third party only + unit: /1 +GOVERNMENT_ONLY: + description: Share of UC households with deductions with only government debt deductions. + values: + 2015-04-01: 0.227 + metadata: + label: UC deduction combination share, government only + unit: /1 +ADVANCE_AND_GOVERNMENT: + description: Share of UC households with deductions repaying advances and government debt. + values: + 2015-04-01: 0.180 + metadata: + label: UC deduction combination share, advances and government + unit: /1 +ADVANCE_AND_THIRD_PARTY: + description: Share of UC households with deductions repaying advances with third party deductions. + values: + 2015-04-01: 0.100 + metadata: + label: UC deduction combination share, advances and third party + unit: /1 +THIRD_PARTY_AND_GOVERNMENT: + description: Share of UC households with deductions with third party and government debt deductions. + values: + 2015-04-01: 0.030 + metadata: + label: UC deduction combination share, third party and government + unit: /1 +ALL_THREE: + description: Share of UC households with deductions with all three deduction types. + values: + 2015-04-01: 0.080 + metadata: + label: UC deduction combination share, all three types + unit: /1 +metadata: + label: UC deduction type combination distribution + reference: + - title: DWP, Universal Credit deductions statistics March 2025 to February 2026, supplementary data tables, Table 5 + href: https://assets.publishing.service.gov.uk/media/69fb3ea22a6137e93226b7ce/universal-credit-deductions-march-2025-to-february-2026.ods diff --git a/policyengine_uk/tests/microsimulation/test_uc_deductions_aggregates.py b/policyengine_uk/tests/microsimulation/test_uc_deductions_aggregates.py new file mode 100644 index 000000000..fb299602d --- /dev/null +++ b/policyengine_uk/tests/microsimulation/test_uc_deductions_aggregates.py @@ -0,0 +1,57 @@ +"""UC deductions per-household statistics against DWP deductions statistics. + +Reference: DWP, Universal Credit deductions statistics March 2025 to February +2026 (published 12 May 2026). Tolerances are wide enough to absorb dataset +revisions but catch structural breakage. Aggregate levels (total deducted, +counts) scale with the model's UC caseload and are not asserted here. +""" + +import numpy as np +import pytest + +from policyengine_uk import Microsimulation + + +@pytest.mark.microsimulation +def test_deduction_statistics_match_dwp(): + sim = Microsimulation() + + for year, cap, at_cap_range, mean_range in [ + # Pre-FRR regime: DWP observed incidence .47, at-25%-cap .13-.14, + # mean monthly deduction 67-68 GBP (March-May 2025). + (2024, 0.25, (0.09, 0.18), (55, 80)), + # FRR regime: DWP observed incidence .46, at-15%-cap .21, + # mean monthly deduction 51-54 GBP (June 2025-February 2026). The + # latent-demand model predicts a somewhat higher at-cap share than + # observed (post-FRR attrition), so the band is wider above. + (2025, 0.15, (0.17, 0.30), (40, 65)), + ]: + w = sim.calculate("benunit_weight", year).values + uc = sim.calculate("universal_credit", year).values + has = sim.calculate("uc_has_deduction", year).values + rate = sim.calculate("uc_deduction_rate", year).values + deductions = sim.calculate("uc_deductions", year).values + + on_uc = uc > 0 + uc_weight = w[on_uc].sum() + deducting_weight = w[on_uc & has].sum() + + incidence = deducting_weight / uc_weight + assert 0.40 < incidence < 0.52, (year, incidence) + + at_cap = w[on_uc & (np.abs(rate - cap) < 1e-4)].sum() / uc_weight + assert at_cap_range[0] < at_cap < at_cap_range[1], (year, at_cap) + + mean_monthly = deductions[on_uc & has] @ w[on_uc & has] / deducting_weight / 12 + assert mean_range[0] < mean_monthly < mean_range[1], ( + year, + mean_monthly, + ) + + +@pytest.mark.microsimulation +def test_draws_are_dispersed_in_microdata(): + sim = Microsimulation() + draws = sim.calculate("uc_deduction_random_draw", 2025).values + assert draws.std() > 0.2 + assert 0.45 < draws.mean() < 0.55 diff --git a/policyengine_uk/tests/policy/baseline/finance/benefit/family/universal_credit/uc_deductions.yaml b/policyengine_uk/tests/policy/baseline/finance/benefit/family/universal_credit/uc_deductions.yaml new file mode 100644 index 000000000..1840c1391 --- /dev/null +++ b/policyengine_uk/tests/policy/baseline/finance/benefit/family/universal_credit/uc_deductions.yaml @@ -0,0 +1,134 @@ +- name: Households have no deductions by default in single-household simulations + period: 2025 + absolute_error_margin: 0.01 + input: + people: + person: + age: 30 + benunits: + benunit: + members: person + would_claim_uc: true + universal_credit_pre_benefit_cap: 6_000 + benefit_cap_reduction: 0 + households: + household: + members: person + output: + uc_deductions: 0 + universal_credit: 6_000 + +- name: Deductions reduce the UC award under the 25% cap in 2024 + period: 2024 + absolute_error_margin: 0.01 + input: + people: + person: + age: 30 + benunits: + benunit: + members: person + would_claim_uc: true + universal_credit_pre_benefit_cap: 6_000 + benefit_cap_reduction: 0 + uc_latent_deduction_rate: 0.25 + uc_deduction_combination: ADVANCE_ONLY + households: + household: + members: person + output: + # Single claimant 25 or over: standard allowance 393.45/month in 2024-25. + # 0.25 x 393.45 x 12 = 1,180.35. + uc_deductions: 1_180.35 + universal_credit: 4_819.65 + +- name: The Fair Repayment Rate caps the same latent demand at 15% in 2025 + period: 2025 + absolute_error_margin: 0.01 + input: + people: + person: + age: 30 + benunits: + benunit: + members: person + would_claim_uc: true + universal_credit_pre_benefit_cap: 6_000 + benefit_cap_reduction: 0 + uc_latent_deduction_rate: 0.25 + uc_deduction_combination: ADVANCE_ONLY + households: + household: + members: person + output: + # Standard allowance 400.14/month in 2025-26. 0.15 x 400.14 x 12 = 720.25. + uc_deductions: 720.25 + universal_credit: 5_279.75 + +- name: The last resort excess above 25% sits on top of the cap + period: 2025 + absolute_error_margin: 0.01 + input: + people: + person: + age: 30 + benunits: + benunit: + members: person + would_claim_uc: true + universal_credit_pre_benefit_cap: 6_000 + benefit_cap_reduction: 0 + uc_latent_deduction_rate: 0.28 + uc_deduction_combination: GOVERNMENT_ONLY + households: + household: + members: person + output: + # Cappable portion min(0.25, cap 0.15) plus last resort excess 0.03: + # 0.18 x 400.14 x 12 = 864.30. + uc_deductions: 864.30 + +- name: Deductions leave at least one penny per assessment period payable + period: 2025 + absolute_error_margin: 0.01 + input: + people: + person: + age: 30 + benunits: + benunit: + members: person + would_claim_uc: true + universal_credit_pre_benefit_cap: 500 + benefit_cap_reduction: 0 + uc_latent_deduction_rate: 0.25 + uc_deduction_combination: ADVANCE_ONLY + households: + household: + members: person + output: + # Schedule 6 of SI 2013/380: the award cannot be deducted below one + # penny per assessment period (12p per year). + uc_deductions: 499.88 + universal_credit: 0.12 + +- name: High-incidence regions still get no deductions by default in single-household simulations + period: 2025 + absolute_error_margin: 0.01 + input: + people: + person: + age: 30 + benunits: + benunit: + members: person + would_claim_uc: true + universal_credit_pre_benefit_cap: 6_000 + benefit_cap_reduction: 0 + households: + household: + members: person + region: NORTH_EAST + output: + uc_deductions: 0 + universal_credit: 6_000 diff --git a/policyengine_uk/tests/test_uc_deductions.py b/policyengine_uk/tests/test_uc_deductions.py new file mode 100644 index 000000000..ffab86dfe --- /dev/null +++ b/policyengine_uk/tests/test_uc_deductions.py @@ -0,0 +1,237 @@ +"""Tests for Universal Credit deduction assignment and reform levers.""" + +import numpy as np +import pytest +from policyengine_core.enums import EnumArray + +from policyengine_uk import Simulation +from policyengine_uk.utils.stochastic import splitmix64_uniform + +YEAR = 2025 + + +def make_situation(**benunit_overrides): + benunit = { + "members": ["person"], + "would_claim_uc": {YEAR: True}, + "universal_credit_pre_benefit_cap": {YEAR: 6_000}, + "benefit_cap_reduction": {YEAR: 0}, + **benunit_overrides, + } + return { + "people": {"person": {"age": {YEAR: 30}}}, + "benunits": {"benunit": benunit}, + "households": {"household": {"members": ["person"]}}, + } + + +class TestSplitmixUniform: + def test_deterministic(self): + ids = np.arange(10_000) + assert np.array_equal(splitmix64_uniform(ids), splitmix64_uniform(ids)) + + def test_uniform_on_unit_interval(self): + draws = splitmix64_uniform(np.arange(100_000)) + assert draws.min() >= 0 + assert draws.max() < 1 + assert abs(draws.mean() - 0.5) < 0.01 + # Deciles should each hold ~10% of draws. + counts = np.histogram(draws, bins=10, range=(0, 1))[0] + assert np.all(abs(counts / 100_000 - 0.1) < 0.01) + + def test_salts_give_independent_streams(self): + ids = np.arange(100_000) + a = splitmix64_uniform(ids, salt=0) + b = splitmix64_uniform(ids, salt=1) + assert abs(np.corrcoef(a, b)[0, 1]) < 0.01 + + +class TestCalculatorMode: + def test_no_deductions_by_default(self): + sim = Simulation(situation=make_situation()) + assert sim.calculate("uc_deductions", YEAR)[0] == 0 + + def test_draws_default_to_one(self): + # 1.0 never falls below any incidence, so calculators get no + # deductions in any region unless set explicitly. + sim = Simulation(situation=make_situation()) + assert sim.calculate("uc_deduction_random_draw", YEAR)[0] == 1.0 + + +class TestReformLevers: + def test_raising_the_cap_restores_pre_frr_deductions(self): + situation = make_situation( + uc_latent_deduction_rate={YEAR: 0.25}, + uc_deduction_combination={YEAR: "ADVANCE_ONLY"}, + ) + baseline = Simulation(situation=situation) + reform = Simulation( + situation=situation, + reform={ + "gov.dwp.universal_credit.deductions.cap": { + "2025-01-01.2025-12-31": 0.25 + } + }, + ) + sa = baseline.calculate("uc_standard_allowance", YEAR)[0] + assert baseline.calculate("uc_deductions", YEAR)[0] == pytest.approx(0.15 * sa) + assert reform.calculate("uc_deductions", YEAR)[0] == pytest.approx(0.25 * sa) + + def test_abolishing_a_type_scales_by_its_amount_share(self): + situation = make_situation( + uc_latent_deduction_rate={YEAR: 0.10}, + uc_deduction_combination={YEAR: "ADVANCE_AND_GOVERNMENT"}, + ) + baseline = Simulation(situation=situation) + reform = Simulation( + situation=situation, + reform={ + "gov.dwp.universal_credit.deductions.abolish.government": { + "2025-01-01.2025-12-31": True + } + }, + ) + sa = baseline.calculate("uc_standard_allowance", YEAR)[0] + assert baseline.calculate("uc_deductions", YEAR)[0] == pytest.approx(0.10 * sa) + # Advances are 41/(41+59) of this combination's deductions. + assert reform.calculate("uc_deductions", YEAR)[0] == pytest.approx( + 0.10 * (41 / 100) * sa + ) + + def test_protected_floor_limits_combined_reductions(self): + # The JRF-style case: the benefit cap plus deductions far exceed 15% + # of the standard allowance; the floor limits the combined + # reductions to (1 - 0.85) x standard allowance. + situation = make_situation( + benefit_cap_reduction={YEAR: 2_500}, + uc_latent_deduction_rate={YEAR: 0.25}, + uc_deduction_combination={YEAR: "ADVANCE_ONLY"}, + ) + baseline = Simulation(situation=situation) + reform = Simulation( + situation=situation, + reform={ + "gov.dwp.universal_credit.deductions.protected_floor": { + "2025-01-01.2025-12-31": 0.85 + } + }, + ) + sa = baseline.calculate("uc_standard_allowance", YEAR)[0] + deductions = baseline.calculate("uc_deductions", YEAR)[0] + assert baseline.calculate("universal_credit", YEAR)[0] == pytest.approx( + 6_000 - 2_500 - deductions + ) + assert reform.calculate("universal_credit", YEAR)[0] == pytest.approx( + 6_000 - 0.15 * sa + ) + + def test_zeroing_a_type_mean_does_not_crash(self): + situation = make_situation( + uc_latent_deduction_rate={YEAR: 0.10}, + uc_deduction_combination={YEAR: "ADVANCE_ONLY"}, + ) + reform = Simulation( + situation=situation, + reform={ + "gov.simulation.uc_deductions.mean_monthly_amount_by_type.ADVANCE": { + "2025-01-01.2025-12-31": 0 + } + }, + ) + assert reform.calculate("uc_deductions", YEAR)[0] == 0 + + def test_protected_floor_inactive_when_reductions_stay_above_it(self): + # Deductions alone under the 15% cap leave the award above an 85% + # floor here, so the floor changes nothing. + situation = make_situation( + uc_latent_deduction_rate={YEAR: 0.25}, + uc_deduction_combination={YEAR: "ADVANCE_ONLY"}, + ) + baseline = Simulation(situation=situation) + reform = Simulation( + situation=situation, + reform={ + "gov.dwp.universal_credit.deductions.protected_floor": { + "2025-01-01.2025-12-31": 0.85 + } + }, + ) + assert reform.calculate("universal_credit", YEAR)[0] == pytest.approx( + baseline.calculate("universal_credit", YEAR)[0] + ) + + def test_protected_floor_binds_on_the_above_cap_excess(self): + # Latent demand of 30% is 15% cappable plus a 15% last resort excess + # that current law exempts from the cap, giving a 20% deduction rate + # under the 15% cap. The modeled floor limits combined reductions + # regardless of category, so it cuts into that excess. JRF's briefing + # does not say whether their floor exempts last resort and child + # maintenance deductions; this pins the modeling choice. + situation = make_situation( + uc_latent_deduction_rate={YEAR: 0.30}, + uc_deduction_combination={YEAR: "ADVANCE_ONLY"}, + ) + baseline = Simulation(situation=situation) + reform = Simulation( + situation=situation, + reform={ + "gov.dwp.universal_credit.deductions.protected_floor": { + "2025-01-01.2025-12-31": 0.85 + } + }, + ) + sa = baseline.calculate("uc_standard_allowance", YEAR)[0] + assert baseline.calculate("uc_deduction_rate", YEAR)[0] == pytest.approx(0.20) + assert reform.calculate("universal_credit", YEAR)[0] == pytest.approx( + 6_000 - 0.15 * sa + ) + + def test_zeroing_every_type_share_leaves_a_valid_enum_array(self): + situation = make_situation( + uc_deduction_random_draw={YEAR: 0.0}, + uc_latent_deduction_rate={YEAR: 0.10}, + ) + reform = Simulation( + situation=situation, + reform={ + f"gov.simulation.uc_deductions.type_combination.{key}": { + "2025-01-01.2025-12-31": 0 + } + for key in [ + "ADVANCE_ONLY", + "THIRD_PARTY_ONLY", + "GOVERNMENT_ONLY", + "ADVANCE_AND_GOVERNMENT", + "ADVANCE_AND_THIRD_PARTY", + "THIRD_PARTY_AND_GOVERNMENT", + "ALL_THREE", + ] + }, + ) + assert reform.calculate("uc_deduction_combination", YEAR)[0] == "NONE" + # The zero-total branch must encode like the normal branch, so + # downstream enum comparisons still work. + stored = reform.get_holder("uc_deduction_combination").get_array(str(YEAR)) + assert isinstance(stored, EnumArray) + assert reform.calculate("uc_deductions", YEAR)[0] == 0 + + def test_abolishing_all_types_removes_deductions(self): + situation = make_situation( + uc_latent_deduction_rate={YEAR: 0.25}, + uc_deduction_combination={YEAR: "ALL_THREE"}, + ) + reform = Simulation( + situation=situation, + reform={ + "gov.dwp.universal_credit.deductions.abolish.advance": { + "2025-01-01.2025-12-31": True + }, + "gov.dwp.universal_credit.deductions.abolish.third_party": { + "2025-01-01.2025-12-31": True + }, + "gov.dwp.universal_credit.deductions.abolish.government": { + "2025-01-01.2025-12-31": True + }, + }, + ) + assert reform.calculate("uc_deductions", YEAR)[0] == 0 diff --git a/policyengine_uk/utils/stochastic.py b/policyengine_uk/utils/stochastic.py new file mode 100644 index 000000000..4fcdc6174 --- /dev/null +++ b/policyengine_uk/utils/stochastic.py @@ -0,0 +1,27 @@ +"""Deterministic pseudo-random draws for stochastic variable assignment. + +Draws are pure functions of entity ids, so results are reproducible across +runs and machines, and datasets can override them by providing the draw +variable directly. +""" + +import numpy as np + + +def splitmix64_uniform(ids: np.ndarray, salt: int = 0) -> np.ndarray: + """Map integer ids to deterministic uniform draws on [0, 1). + + Uses the splitmix64 finalizer, which passes standard statistical tests + for avalanche behavior. Different salts give independent streams. + """ + with np.errstate(over="ignore"): + z = ids.astype(np.uint64) + np.uint64(salt) * np.uint64(0x632BE59BD9B4E019) + z = z + np.uint64(0x9E3779B97F4A7C15) + z = (z ^ (z >> np.uint64(30))) * np.uint64(0xBF58476D1CE4E5B9) + z = (z ^ (z >> np.uint64(27))) * np.uint64(0x94D049BB133111EB) + z = z ^ (z >> np.uint64(31)) + # Use the top 53 bits so the result is exactly representable, and clamp + # below the largest float32 under 1: model variables store as float32, + # which would otherwise round near-1 draws up to exactly 1.0. + draws = (z >> np.uint64(11)).astype(np.float64) / 2.0**53 + return np.minimum(draws, 1.0 - 2.0**-24) diff --git a/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_deduction_combination.py b/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_deduction_combination.py new file mode 100644 index 000000000..c76ca38f3 --- /dev/null +++ b/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_deduction_combination.py @@ -0,0 +1,68 @@ +from policyengine_uk.model_api import * + + +class UCDeductionCombination(Enum): + NONE = "No deductions" + ADVANCE_ONLY = "Advance deductions only" + THIRD_PARTY_ONLY = "Third party deductions only" + GOVERNMENT_ONLY = "Government deductions only" + ADVANCE_AND_GOVERNMENT = "Advance and government deductions" + ADVANCE_AND_THIRD_PARTY = "Advance and third party deductions" + THIRD_PARTY_AND_GOVERNMENT = "Third party and government deductions" + ALL_THREE = "Advance, third party and government deductions" + + +class uc_deduction_combination(Variable): + label = "UC deduction type combination" + documentation = ( + "Which deduction types this benefit unit repays, assigned from the " + "DWP-published distribution of type combinations among UC households " + "with deductions." + ) + entity = BenUnit + definition_period = YEAR + value_type = Enum + possible_values = UCDeductionCombination + default_value = UCDeductionCombination.NONE + + def formula(benunit, period, parameters): + # Estimation fallback: datasets can impute this directly (or override + # the draw); the parameter lives under gov.simulation because it + # describes the world, not the law. + c = parameters(period).gov.simulation.uc_deductions.type_combination + shares = np.array( + [ + c.ADVANCE_ONLY, + c.THIRD_PARTY_ONLY, + c.GOVERNMENT_ONLY, + c.ADVANCE_AND_GOVERNMENT, + c.ADVANCE_AND_THIRD_PARTY, + c.THIRD_PARTY_AND_GOVERNMENT, + c.ALL_THREE, + ] + ) + total = shares.sum() + if total <= 0: + # Every share reformed to zero: nobody is assigned a combination. + # filled_array keeps this branch on the same encoding path as the + # select below, so it yields an EnumArray like the normal branch. + return benunit.filled_array(UCDeductionCombination.NONE) + cumulative = np.cumsum(shares / total) + draw = benunit("uc_deduction_type_random_draw", period) + index = np.searchsorted(cumulative, clip(draw, 0, 1 - 1e-9), side="right") + index = clip(index, 0, len(shares) - 1) + has_deduction = benunit("uc_has_deduction", period) + combinations = [ + UCDeductionCombination.ADVANCE_ONLY, + UCDeductionCombination.THIRD_PARTY_ONLY, + UCDeductionCombination.GOVERNMENT_ONLY, + UCDeductionCombination.ADVANCE_AND_GOVERNMENT, + UCDeductionCombination.ADVANCE_AND_THIRD_PARTY, + UCDeductionCombination.THIRD_PARTY_AND_GOVERNMENT, + UCDeductionCombination.ALL_THREE, + ] + return select( + [has_deduction & (index == i) for i in range(len(combinations))], + combinations, + default=UCDeductionCombination.NONE, + ) diff --git a/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_deduction_random_draw.py b/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_deduction_random_draw.py new file mode 100644 index 000000000..22c73c1c3 --- /dev/null +++ b/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_deduction_random_draw.py @@ -0,0 +1,26 @@ +from policyengine_uk.model_api import * +from policyengine_uk.utils.stochastic import splitmix64_uniform + + +class uc_deduction_random_draw(Variable): + label = "UC deduction random draw" + documentation = ( + "Uniform draw on [0, 1) determining deduction incidence and size. " + "Deterministic hash of the benefit unit id in dataset simulations; " + "1.0 in single-household simulations, so no deduction unless set. " + "Datasets and situations can override it directly." + ) + entity = BenUnit + definition_period = YEAR + value_type = float + default_value = 1.0 + + def formula(benunit, period, parameters): + # Representative microdata carries tens of millions of households of + # weight; single-household situations carry ~1. Only assign hashed + # draws in representative simulations: the 1.0 default never falls + # below any incidence, so calculators get no deductions unless set. + if benunit("benunit_weight", period).sum() < 1e6: + return np.ones(benunit.count) + ids = benunit("benunit_id", period) + return splitmix64_uniform(ids, salt=0) diff --git a/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_deduction_rate.py b/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_deduction_rate.py new file mode 100644 index 000000000..8271d6053 --- /dev/null +++ b/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_deduction_rate.py @@ -0,0 +1,80 @@ +from policyengine_uk.model_api import * +from policyengine_uk.variables.gov.dwp.universal_credit.deductions.uc_deduction_combination import ( + UCDeductionCombination, +) + + +class uc_deduction_rate(Variable): + label = "UC deduction rate" + documentation = ( + "Deductions taken from this benefit unit's Universal Credit award as " + "a fraction of the standard allowance, after removing any abolished " + "deduction types and applying the deductions cap. The above-cap " + "excess (latent demand above 25% of the standard allowance, covering " + "last resort and child maintenance deductions) sits on top of the " + "cap." + ) + entity = BenUnit + definition_period = YEAR + value_type = float + + def formula(benunit, period, parameters): + # Law: the cap and abolition switches. Estimation: the mean amounts + # by type (gov.simulation), used only to split a household's total + # rate across its deduction types. + law = parameters(period).gov.dwp.universal_credit.deductions + m = parameters(period).gov.simulation.uc_deductions.mean_monthly_amount_by_type + latent = benunit("uc_latent_deduction_rate", period) + combination = benunit("uc_deduction_combination", period) + + advance_weight = 0.0 if law.abolish.advance else m.ADVANCE + third_party_weight = 0.0 if law.abolish.third_party else m.THIRD_PARTY + government_weight = 0.0 if law.abolish.government else m.GOVERNMENT + + def retained(kept, total): + return kept / max(total, 1e-9) + + retained_share = select( + [ + combination == UCDeductionCombination.ADVANCE_ONLY, + combination == UCDeductionCombination.THIRD_PARTY_ONLY, + combination == UCDeductionCombination.GOVERNMENT_ONLY, + combination == UCDeductionCombination.ADVANCE_AND_GOVERNMENT, + combination == UCDeductionCombination.ADVANCE_AND_THIRD_PARTY, + combination == UCDeductionCombination.THIRD_PARTY_AND_GOVERNMENT, + combination == UCDeductionCombination.ALL_THREE, + ], + [ + retained(advance_weight, m.ADVANCE), + retained(third_party_weight, m.THIRD_PARTY), + retained(government_weight, m.GOVERNMENT), + retained( + advance_weight + government_weight, + m.ADVANCE + m.GOVERNMENT, + ), + retained( + advance_weight + third_party_weight, + m.ADVANCE + m.THIRD_PARTY, + ), + retained( + third_party_weight + government_weight, + m.THIRD_PARTY + m.GOVERNMENT, + ), + retained( + advance_weight + third_party_weight + government_weight, + m.ADVANCE + m.THIRD_PARTY + m.GOVERNMENT, + ), + ], + default=0.0, + ) + # The cappable portion covers the three modeled deduction types + # (advances, third party, government debt) and responds to the cap + # and abolition switches. Latent demand above the calibration-window + # cap belongs to categories permitted to exceed the cap (last resort + # and child maintenance deductions), exempt from both. + calibration_cap = parameters( + period + ).gov.simulation.uc_deductions.calibration_cap + cappable = min_(latent, calibration_cap) * retained_share + above_cap_excess = max_(latent - calibration_cap, 0) + return min_(cappable, law.cap) + above_cap_excess diff --git a/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_deduction_type_random_draw.py b/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_deduction_type_random_draw.py new file mode 100644 index 000000000..284739d13 --- /dev/null +++ b/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_deduction_type_random_draw.py @@ -0,0 +1,26 @@ +from policyengine_uk.model_api import * +from policyengine_uk.utils.stochastic import splitmix64_uniform + + +class uc_deduction_type_random_draw(Variable): + label = "UC deduction type random draw" + documentation = ( + "Uniform draw on [0, 1) determining the deduction type combination. " + "Deterministic hash of the benefit unit id in dataset simulations; " + "1.0 in single-household simulations. Datasets and situations can " + "override it directly." + ) + entity = BenUnit + definition_period = YEAR + value_type = float + default_value = 1.0 + + def formula(benunit, period, parameters): + # Representative microdata carries tens of millions of households of + # weight; single-household situations carry ~1. Only assign hashed + # draws in representative simulations; 1.0 maps to the last type + # combination but only matters when a deduction is assigned. + if benunit("benunit_weight", period).sum() < 1e6: + return np.ones(benunit.count) + ids = benunit("benunit_id", period) + return splitmix64_uniform(ids, salt=1) diff --git a/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_deductions.py b/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_deductions.py new file mode 100644 index 000000000..6425ae533 --- /dev/null +++ b/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_deductions.py @@ -0,0 +1,31 @@ +from policyengine_uk.model_api import * + + +class uc_deductions(Variable): + label = "UC deductions" + documentation = ( + "Money deducted from this benefit unit's Universal Credit award to " + "repay debts: advance repayments, government debt (benefit and tax " + "credit overpayments) and third party deductions. Deductions leave " + "at least one penny per assessment period payable (Schedule 6 of " + "SI 2013/380). Per-household statistics are validated against the " + "DWP deductions statistics; weighted aggregates are not, because " + "they scale with the model's UC caseload, which falls short of the " + "administrative count." + ) + entity = BenUnit + definition_period = YEAR + value_type = float + unit = GBP + + def formula(benunit, period, parameters): + p = parameters(period).gov.dwp.universal_credit.deductions + rate = benunit("uc_deduction_rate", period) + standard_allowance = benunit("uc_standard_allowance", period) + award = max_( + benunit("universal_credit_pre_benefit_cap", period) + - benunit("benefit_cap_reduction", period), + 0, + ) + minimum_payable = p.minimum_payable * MONTHS_IN_YEAR + return min_(rate * standard_allowance, max_(award - minimum_payable, 0)) diff --git a/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_has_deduction.py b/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_has_deduction.py new file mode 100644 index 000000000..10af0ade4 --- /dev/null +++ b/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_has_deduction.py @@ -0,0 +1,44 @@ +from policyengine_uk.model_api import * + + +class uc_has_deduction(Variable): + label = "UC household has deductions" + documentation = ( + "Whether this benefit unit has money deducted from its Universal " + "Credit award to repay debts (advances, government debt or third " + "party deductions). Assigned from the DWP-published incidence of " + "deductions among UC households, adjusted by region." + ) + entity = BenUnit + definition_period = YEAR + value_type = bool + + def formula(benunit, period, parameters): + # Estimation fallback: datasets can impute this directly (or override + # the draw); the parameters live under gov.simulation because they + # describe the world, not the law. + p = parameters(period).gov.simulation.uc_deductions + d = p.latent_rate_distribution + incidence = ( + d.UNDER_5 + + d.AT_5 + + d.FIVE_TO_10 + + d.AT_10 + + d.TEN_TO_15 + + d.AT_15 + + d.FIFTEEN_TO_20 + + d.AT_20 + + d.TWENTY_TO_25 + + d.AT_25 + + d.OVER_25 + ) + region = benunit.household("region", period) + adjusted_incidence = clip(incidence * p.region_incidence_factor[region], 0, 1) + award = max_( + benunit("universal_credit_pre_benefit_cap", period) + - benunit("benefit_cap_reduction", period), + 0, + ) + on_uc = benunit("would_claim_uc", period) & (award > 0) + draw = benunit("uc_deduction_random_draw", period) + return on_uc & (draw < adjusted_incidence) diff --git a/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_latent_deduction_rate.py b/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_latent_deduction_rate.py new file mode 100644 index 000000000..e95bdfc1c --- /dev/null +++ b/policyengine_uk/variables/gov/dwp/universal_credit/deductions/uc_latent_deduction_rate.py @@ -0,0 +1,74 @@ +from policyengine_uk.model_api import * + +# Band edges of the DWP deductions distribution (Table 2 of the deductions +# supplementary tables), as fractions of the standard allowance. Bands where +# the lower and upper edges are equal are point masses at institutional rates. +# The final band holds above-cap deductions (last resort and child +# maintenance categories); the DWP tables top-code it, so 30% is a modeling +# assumption for its upper edge. +BAND_LOWER = np.array([0.0, 0.05, 0.05, 0.10, 0.10, 0.15, 0.15, 0.20, 0.20, 0.25, 0.25]) +BAND_UPPER = np.array( + [0.05, 0.05, 0.10, 0.10, 0.15, 0.15, 0.20, 0.20, 0.25, 0.25, 0.30] +) + + +class uc_latent_deduction_rate(Variable): + label = "UC latent deduction rate" + documentation = ( + "Deduction demand as a fraction of the Universal Credit standard " + "allowance, before applying the statutory cap. Assigned by inverting " + "the DWP-published distribution of deduction rates observed under " + "the 25% cap (March to May 2025), so the cap parameter reproduces " + "observed post-cap distributions and reforms to the cap rescale " + "coherently. Validated for 2024 and 2025; earlier years apply the " + "same distribution under the higher caps then in force. Demand is " + "held fixed as the cap moves, so cuts below the 15% cap overstate " + "the constrained population while cap increases toward 25% stay " + "inside the observed window." + ) + entity = BenUnit + definition_period = YEAR + value_type = float + + def formula(benunit, period, parameters): + # Estimation fallback: datasets can impute this directly (or override + # the draw); the parameters live under gov.simulation because they + # describe the world, not the law. + p = parameters(period).gov.simulation.uc_deductions + d = p.latent_rate_distribution + shares = np.array( + [ + d.UNDER_5, + d.AT_5, + d.FIVE_TO_10, + d.AT_10, + d.TEN_TO_15, + d.AT_15, + d.FIFTEEN_TO_20, + d.AT_20, + d.TWENTY_TO_25, + d.AT_25, + d.OVER_25, + ] + ) + total_share = shares.sum() + if total_share <= 0: + return np.zeros(benunit.count) + has_deduction = benunit("uc_has_deduction", period) + draw = benunit("uc_deduction_random_draw", period) + region = benunit.household("region", period) + adjusted_incidence = clip( + total_share * p.region_incidence_factor[region], 1e-9, 1 + ) + # Conditional on assignment (draw < adjusted incidence), the rescaled + # draw is uniform on [0, 1): use it as the quantile of the rate + # distribution. + quantile = clip(draw / adjusted_incidence, 0, 1 - 1e-9) + position = quantile * total_share + cumulative = np.cumsum(shares) + band = np.searchsorted(cumulative, position, side="right") + band = clip(band, 0, len(shares) - 1) + cumulative_before = cumulative[band] - shares[band] + within = (position - cumulative_before) / np.maximum(shares[band], 1e-12) + rate = BAND_LOWER[band] + (BAND_UPPER[band] - BAND_LOWER[band]) * within + return where(has_deduction, rate, 0) diff --git a/policyengine_uk/variables/gov/dwp/universal_credit/universal_credit.py b/policyengine_uk/variables/gov/dwp/universal_credit/universal_credit.py index 41fdd777a..f30621334 100644 --- a/policyengine_uk/variables/gov/dwp/universal_credit/universal_credit.py +++ b/policyengine_uk/variables/gov/dwp/universal_credit/universal_credit.py @@ -12,4 +12,25 @@ class universal_credit(Variable): def formula(benunit, period, parameters): uc_max_entitlement = benunit("universal_credit_pre_benefit_cap", period) benefit_cap_reduction = benunit("benefit_cap_reduction", period) - return max_(uc_max_entitlement - benefit_cap_reduction, 0) + deductions = benunit("uc_deductions", period) + # A protected minimum floor (zero under current law) caps combined + # benefit cap reductions and deductions at (1 - floor) x the standard + # allowance, per JRF's protected minimum floor design (their worked + # example limits the reduction itself to 15% of the standard + # allowance). Zero means reductions are unlimited. + # The floor binds on the whole of uc_deductions, including the above- + # cap excess that current law exempts from the deductions cap (last + # resort and child maintenance deductions). JRF's briefing does not + # say whether their floor exempts those categories - its worked + # example involves only cappable deductions and the benefit cap - so + # this is a modeling choice, tracked for follow-up. + floor_rate = parameters( + period + ).gov.dwp.universal_credit.deductions.protected_floor + total_reductions = benefit_cap_reduction + deductions + max_reductions = where( + floor_rate > 0, + (1 - floor_rate) * benunit("uc_standard_allowance", period), + np.inf, + ) + return max_(uc_max_entitlement - min_(total_reductions, max_reductions), 0) diff --git a/uv.lock b/uv.lock index 15b41ec1a..a1db86a16 100644 --- a/uv.lock +++ b/uv.lock @@ -1584,7 +1584,7 @@ wheels = [ [[package]] name = "policyengine-uk" -version = "2.88.22" +version = "2.89.4" source = { editable = "." } dependencies = [ { name = "microdf-python", marker = "python_full_version >= '3.11'" }, @@ -1598,7 +1598,6 @@ dev = [ { name = "coverage", marker = "python_full_version >= '3.11'" }, { name = "furo", marker = "python_full_version >= '3.11'" }, { name = "jupyter-book", marker = "python_full_version >= '3.11'" }, - { name = "pytest", marker = "python_full_version >= '3.11'" }, { name = "pytest-cov", marker = "python_full_version >= '3.11'" }, { name = "rich", marker = "python_full_version >= '3.11'" }, { name = "ruff", marker = "python_full_version >= '3.11'" }, @@ -1608,7 +1607,6 @@ dev = [ { name = "sphinx-math-dollar", marker = "python_full_version >= '3.11'" }, { name = "towncrier", marker = "python_full_version >= '3.11'" }, { name = "tqdm", marker = "python_full_version >= '3.11'" }, - { name = "wheel", marker = "python_full_version >= '3.11'" }, ] [package.metadata] @@ -1619,7 +1617,6 @@ requires-dist = [ { name = "microdf-python", specifier = ">=1.2.1" }, { name = "policyengine-core", specifier = ">=3.26.0" }, { name = "pydantic", specifier = ">=2.11.7" }, - { name = "pytest", marker = "extra == 'dev'" }, { name = "pytest-cov", marker = "extra == 'dev'" }, { name = "rich", marker = "extra == 'dev'" }, { name = "ruff", marker = "extra == 'dev'", specifier = ">=0.9.0" }, @@ -1632,7 +1629,6 @@ requires-dist = [ { name = "tables", marker = "python_full_version >= '3.11'", specifier = ">=3.10.2" }, { name = "towncrier", marker = "extra == 'dev'", specifier = ">=24.8.0" }, { name = "tqdm", marker = "extra == 'dev'" }, - { name = "wheel", marker = "extra == 'dev'" }, ] provides-extras = ["dev"]